Retimers Extend Die-to-Die Interconnects

Abstract

The retimer includes a first port that couples to a die via a first interconnect, the first interconnect including a defined set of lanes and utilizing a first communications technology, the die being disposed on a first package with the retimer, the retimer further includes a second port that couples to another retimer via a second interconnect, the second interconnect utilizing a different second communications technology, the second retimer being disposed on a different second package, facilitating a longer reach communications channel.

Description

[Technical field]

[0001] Related Applications This application claims priority to U.S. patent application Ser. No. 17 / 855,720, filed on June 30, 2022, and entitled "RETIMER FOR ENHANCED DIE-TO-DIE INTERCONNECT," which claims benefit and priority under 35 U.S.C. Section 119 to Indian Provisional Patent Application No. 202241004437, filed on January 27, 2022, and entitled "ON-PACKAGE DIE-TO-DIE (D2D) RETIMER," both of which are incorporated herein by reference in their entirety.

[0002] The present disclosure relates to computing systems, and more particularly (though not exclusively) to physical interconnects and associated link protocols. [Background technology]

[0003] Advances in semiconductor processing and logic design have enabled an increase in the amount of logic that can reside in an integrated circuit device. As a natural consequence, computer system configurations have evolved from a single or multiple integrated circuits in a system to multiple cores, multiple hardware threads, and multiple logical processors on individual integrated circuits, as well as other interfaces integrated with such processors. A processor or integrated circuit typically comprises a single physical processor die, which may include any number of cores, hardware threads, logical processors, interfaces, memory, controller hubs, etc.

[0004] Smaller computing devices have become more popular as a result of their greater ability to fit more processing power in a smaller package. Smartphones, tablets, ultra-thin notebooks, and other user equipment have grown exponentially. However, these smaller devices rely on servers for both data storage and complex processing that exceeds their form factor. As a result, the need in the high performance computing market (i.e., server space) has also increased. For example, in today's servers, there is typically not only a single processor with multiple cores, but also multiple physical processors (also referred to as multiple sockets) to increase computing power. However, as processing power increases along with the number of devices in a computing system, communication between the sockets and other devices becomes more important.

[0005] In fact, interconnects have grown from traditional multi-drop buses that primarily handled electrical communications to full-blown interconnect architectures that facilitate high-speed communications. Unfortunately, there is a need for future processors that are much more efficiently utilized, and a corresponding demand is being placed on the capabilities of existing interconnect architectures. [Brief description of the drawings]

[0006] [Figure 1] 1 illustrates an embodiment of a computing system that includes an interconnect architecture.

[0007] [Diagram 2] 1 illustrates an embodiment of an interconnect architecture that includes a hierarchical stack.

[0008] [Diagram 3] 1 illustrates an embodiment of a potential high performance processor-to-processor interconnect configuration.

[0009] [Figure 4]1 illustrates an embodiment of a layered protocol stack associated with an interconnect.

[0010] [Diagram 5] 1 shows a simplified block diagram of an exemplary computing system utilizing a Compute Express Link (CXL)-based protocol-compliant link.

[0011] [Figure 6] 1 shows a simplified block diagram of an exemplary system-on-chip (SoC) package.

[0012] [Figure 7] FIG. 1 shows a simplified block diagram of a two-dimensional packaging substrate and interconnects coupling two or more devices.

[0013] [Figure 8A] FIG. 1 shows a simplified block diagram of an exemplary packaging configuration. [Figure 8B] FIG. 1 shows a simplified block diagram of an exemplary packaging configuration. [Figure 8C] FIG. 1 shows a simplified block diagram of an exemplary packaging configuration. [Figure 8D] FIG. 1 shows a simplified block diagram of an exemplary packaging configuration.

[0014] [Figure 9] FIG. 1 is a simplified block diagram illustrating an example rack server having computing devices interconnected using a Universal Chiplet Interconnect Express (UCIe) based protocol.

[0015] [Figure 10] FIG. 1 is a simplified block diagram illustrating an example protocol stack for UCIe.

[0016] [Figure 11]FIG. 1 is a simplified block diagram of a computing system including two packages connected by a UCIe-based interconnect.

[0017] [Figure 12] FIG. 2 is a simplified block diagram of an exemplary retimer device.

[0018] [Figure 13] FIG. 2 is a simplified block diagram of an example physical layer block.

[0019] [Figure 14A] 1 is a representation of an exemplary UCIe data format. [Figure 14B] 1 is a representation of an exemplary UCIe data format. [Figure 14C] 1 is a representation of an exemplary UCIe data format.

[0020] [Figure 15] FIG. 2 is an example signaling diagram of a lane of an example UCIe link.

[0021] [Figure 16A] FIG. 2 is a simplified block diagram of an example communication channel including an external link facilitated by an example retimer device.

[0022] [Figure 16B] 4 is a simplified flow diagram illustrating an example link initialization.

[0023] [Figure 17] 1 is a representation of an exemplary link state machine.

[0024] [Figure 18A] 1 is a representation of an exemplary register for use in an exemplary retimer. [Figure 18B] 1 is a representation of an exemplary register for use in an exemplary retimer. [Figure 18C] 1 is a representation of an exemplary register for use in an exemplary retimer. [Figure 18D] 1 is a representation of an exemplary register for use in an exemplary retimer.

[0025] [Figure 19A] A representation of a data structure for use in facilitating software accessed registers. [Figure 19B] A representation of a data structure for use in facilitating software accessed registers.

[0026] [Figure 20] 1 is a representation of an exemplary register locator register.

[0027] [Figure 21] 1 illustrates an embodiment of a block diagram of a computing system including a multi-core processor.

[0028] [Figure 22] 1 illustrates an embodiment of a block of a computing system including multiple processors. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0029] In the following description, numerous specific details are described to provide a thorough understanding of the present invention, such as examples of specific types of processors and system configurations, specific hardware structures, specific architectural and microarchitectural details, specific register configurations, specific instruction types, specific system components, specific measurements / heights, specific processor pipeline stages, and operations. However, it will be apparent to those skilled in the art that these specific details need not be employed to implement the solutions provided in this disclosure. In other examples, well-known components or methods, such as specific and alternative processor architectures, specific logic circuits / code for the described algorithms, specific firmware code, specific interconnect operations, specific logic configurations, specific manufacturing techniques and materials, specific compiler implementations, specific expressions for algorithms in the code, specific power-down and gating techniques / logic, and other specific operational details of computer systems, have not been described in detail to avoid unnecessarily obscuring the present invention.

[0030] Although the following embodiments may be described with respect to energy management and energy efficiency within a particular integrated circuit, such as a computing platform or a microprocessor, other embodiments are applicable to other types of integrated circuits and logic devices. Similar techniques and teachings of the embodiments described herein may be applied to other types of circuits or semiconductor devices that may also benefit from better energy efficiency and energy management. For example, the disclosed embodiments are not limited to desktop computer systems or Ultrabooks, but may be used in other devices such as handheld devices, tablets, other thin notebooks, system-on-a-chip (SoC) devices, and embedded applications. Some examples of handheld devices include cellular phones, Internet protocol devices, digital cameras, personal digital assistants (PDAs), and handheld PCs. Embedded applications generally include microcontrollers, digital signal processors (DSPs), systems-on-a-chips, network computers (NetPCs), set-top boxes, network hubs, wide area network (WAN) switches, or any other system capable of performing the functions and operations taught below. The techniques and teachings of the embodiments described herein may be applied at the server level, including rack-scale server platforms, blade server platforms, and other server architectures. Additionally, the apparatus, methods, and systems described herein are not limited to physical computing devices, but may relate to software emulation and simulation of physical systems, such as the architectures described in the examples below. As will become readily apparent from the following description, the method, apparatus, and system embodiments described herein (whether relating to hardware, firmware, software, or a combination thereof) are important in enabling computing platforms to process increasing data rates while balancing power utilization and physical footprint, among other exemplary considerations.

[0031] As computing systems advance, the components therein become more complex. As a result, the interconnect architectures for coupling and communicating between components also become more complex to ensure that bandwidth requirements for optimal component operation are met. Furthermore, different market segments require different aspects of the interconnect architecture to meet market needs. For example, servers require higher performance while the mobile ecosystem may in some cases sacrifice overall performance for power savings. However, providing the highest possible performance with maximum power savings is the sole purpose of most fabrics. A number of interconnects described below could potentially benefit from aspects of the solution described herein.

[0032] Modern interconnect fabric architectures enable components and devices from different vendors to interoperate in an open architecture across multiple market segments, such as clients (desktop and mobile), servers (standard and enterprise), and embedded communication devices. For example, high-performance general-purpose I / O interconnect protocols (e.g., Peripheral Component Interconnect Express (PCIe)) are being defined for a wide variety of future computing and communication platforms. Such protocols and corresponding architectures leverage the advantages of point-to-point interconnects, switch-based technologies, and packetized protocols to provide new levels of performance and functionality. As an example, power management, quality of service (QoS), hot-plug / hot-swap support, data integrity, and error handling are some of the advanced features supported by PCI Express.

[0033] Referring to FIG. 1, an embodiment of a fabric composed of point-to-point links interconnecting a set of components is shown. System 100 includes processor 105 and system memory 110 coupled to controller hub 115. Processor 105 includes any processing element such as a microprocessor, host processor, embedded processor, co-processor, or other processor. Processor 105 is coupled to controller hub 115 through front side bus (FSB) 106. In one embodiment, FSB 106 is a serial point-to-point interconnect as described below. In another embodiment, link 106 includes a serial differential interconnect architecture that conforms to a different interconnect standard. The interconnect protocols and functions described below may be utilized to implement the fabric and links that couple the set of components introduced herein in FIG. 1.

[0034] System memory 110 includes any memory device, such as random access memory (RAM), non-volatile (NV) memory, or other memory accessible by multiple devices in system 100. System memory 110 is coupled to controller hub 115 through memory interface 116. Examples of memory interfaces include a double data rate (DDR) memory interface, a dual channel DDR memory interface, and a dynamic RAM (DRAM) memory interface.

[0035] In one embodiment, controller hub 115 may be implemented as a root hub, root complex, or root controller of a PCIe interconnect hierarchy. Examples of controller hub 115 include a chipset, a memory controller hub (MCH), a northbridge, an interconnect controller hub (ICH), a southbridge, and a root controller / hub. The term chipset often refers to two physically separate controller hubs, a memory controller hub (MCH) coupled to an interconnect controller hub (ICH). Note that current systems often include an MCH integrated into processor 105, but controller 115 communicates with I / O devices in a manner similar to that described below. In some embodiments, peer-to-peer routing is optionally supported through root complex 115.

[0036] Here, controller hub 115 is coupled to switch / bridge 120 via serial link 119. Input / output modules 117 and 121, which may also be referred to as interfaces / ports 117 and 121, contain / implement a layered protocol stack to provide communication between controller hub 115 and switch 120. In one embodiment, multiple devices may be coupled to switch 120.

[0037] The switch / bridge 120 routes packets / messages from upstream devices 125, i.e., one level up towards the root complex, to downstream controller hub 115, i.e., one level down away from the root controller, from processor 105 or system memory 110 to devices 125. In one embodiment, the switch 120 is referred to as a logical assembly of multiple virtual PCI-to-PCI bridge devices. Devices 125 include any internal or external device or component coupled to an electronic system, such as I / O devices, network interface controllers (NICs), add-in cards, audio processors, network processors, hard drives, storage devices, CD / DVD ROMs, monitors, printers, mice, keyboards, routers, portable storage devices, Firewire devices, Universal Serial Bus (USB) devices, scanners, and other input / output devices. In PCIe terminology, such devices are often referred to as endpoints. Although not specifically shown, devices 125 may include PCIe-to-PCI / PCI-X bridges to support legacy or other versions of PCI devices. Endpoint devices in PCIe are often categorized as legacy, PCIe or root complex integrated endpoints.

[0038] The graphics accelerator 130 is also coupled to the controller hub 115 via a serial link 132. In one embodiment, the graphics accelerator 130 is coupled to an MCH which is coupled to the ICH. The switch 120, and thus the I / O device 125, are then coupled to the ICH. The I / O modules 131 and 118 also implement a layered protocol stack for communicating between the graphics accelerator 130 and the controller hub 115. As with the MCH described above, the graphics controller or graphics accelerator 130 itself may be integrated into the processor 105. Additionally, one or more links (e.g., 123) of the system may include one or more extension devices (e.g., 150), such as, for example, retimers, repeaters, etc.

[0039] Referring to FIG. 2, an embodiment of a layered protocol stack is shown. The layered protocol stack 200 includes any form of layered communication stack, such as a Quick Path Interconnect (QPI) stack, a PCIe stack, a Compute Express Link (CXL) stack, or other high performance computing interconnect stack, or other layered stack. The immediately following description of FIGS. 1-4 is for a general interconnect protocol stack (e.g., PCIe), but the same concepts may be applied to other interconnect stacks. In one embodiment, the protocol stack 200 is a PCIe protocol stack including a transaction layer 205, a link layer 210, and a physical layer 220. Interfaces such as interfaces 117, 118, 121, 122, 126, and 131 in FIG. 1 may be represented as a communication protocol stack 200. The representation as a communication protocol stack may also be referred to as a module or interface that implements / contains the protocol stack.

[0040] The protocol communicates information between components using packets. Packets are formed in the transaction layer 205 and data link layer 210 to carry information from the sending component to the receiving component. As transmitted packets flow through other layers, they are extended with additional information required for processing the packets at those layers. At the receiving end, reverse processing occurs, converting packets from their physical layer 220 representation to their data link layer 210 representation, and finally (for transaction layer packets) to a form that can be processed by the transaction layer 205 of the receiving device.

[0041] [Transaction Layer]

[0042] In one embodiment, the transaction layer 205 provides an interface between the processing cores of the device and the interconnect architecture, such as the data link layer 210 and the physical layer 220. In this regard, the primary role of the transaction layer 205 is the assembly and disassembly of packets (i.e., transaction layer packets or TLPs). The translation layer 205 typically manages credit-based flow control for TLPs. For example, the protocol may implement split transactions, i.e., transactions with a request and response separated by time, to allow the link to carry other traffic while the target device collects data for the response. In some implementations, the flow control may be implemented using a credit-based flow control scheme. In this scheme, the device provides an initial amount of credit for each of the receive buffers in the transaction layer 205. An external device at the other end of the link, such as the controller hub 115 of FIG. 1, counts the number of credits consumed by each TLP. If the transaction does not exceed the credit limit, the transaction may be sent. Upon receiving the response, the credit amount is restored. The advantage of the credit scheme is that if no credit limits are encountered, the latency of credit return does not impact performance.

[0043] In one embodiment, the four transaction address spaces include a configuration address space, a memory address space, an I / O address space, and a message address space. Memory space transactions include one or more of read and write requests that transfer data to / from memory mapped locations. In one embodiment, memory space transactions can use two different address formats, e.g., a short address format, such as a 32-bit address, or a long address format, such as a 64-bit address. Configuration space transactions are used to access the configuration space of devices that support the protocol. Transactions to the configuration space include read and write requests. Message space transactions (or simply messages) are defined to support in-band communication between protocol agents that implement the protocol on the devices. Thus, in one embodiment, transaction layer 205 constructs packet header / payload 206 .

[0044] [Link Layer]

[0045] The link layer 210, also referred to as the data link layer 210, acts as an intermediary between the transaction layer 205 and the physical layer 220. In one embodiment, the role of the data link layer 210 is to provide a reliable mechanism for exchanging transaction layer packets (TLPs) between two components on a link. One side of the data link layer 210 accepts the TLPs assembled by the transaction layer 205, applies a packet sequence identifier 211, i.e., an identification number or packet number, calculates and applies an error detection code, i.e., CRC 212, and sends the modified TLPs to the physical layer 220 for transmission across the physical device to an external device.

[0046] [Physical layer]

[0047] In one embodiment, physical layer 220 includes logical sub-block 221 and electrical sub-block 222 to physically transmit packets to an external device, where logical sub-block 221 is responsible for the “digital” functions of physical layer 220. In this regard, logical sub-block 221 includes a transmit section for preparing outgoing information for transmission by physical sub-block 222 and a receiver section for identifying and preparing received information before passing it to link layer 210.

[0048] The physical block 222 includes a transmitter and a receiver. The transmitter is provided with symbols by the logical sub-block 221, which serializes and transmits them to a physical transmission medium that connects the transmitter to another external device. The receiver is provided with serialized symbols from the external device and converts the received signal into a bit stream. The bit stream is deserialized and provided to the logical sub-block 221. In one embodiment, an 8b / 10b transmission code is employed, where 10-bit symbols are transmitted / received. In another example, 128b / 130b encoding may be employed, and in yet another implementation, Flit-based transmission may be utilized, among other examples. In one example, a special symbol is used to frame the packets in the frame 223. Additionally, in one example, the receiver also provides a symbol clock recovered from the incoming serial stream.

[0049] The transmission medium may implement a transmission path for transmitting data, such as a transmission line, copper wire, optical line, wireless communication channel, infrared communication link, or other communication path. The connection between two devices is referred to as a link. A link may be implemented with one or more constituent transmission paths, or lanes. To scale the bandwidth, a link may aggregate multiple lanes, denoted by xN, where N is any supported link width, e.g., 1, 2, 4, 8, 12, 16, 32, 64, or more. In some implementations, differential signaling with a differential pair may be employed, which refers to two transmission paths that transmit differential signals. As an example, in a differential pair, when the first line of the pair switches from a low voltage level to a high voltage level, i.e., a rising edge, the other line of the pair drives from a high logic level to a low logic level, i.e., a falling edge. Differential signals potentially exhibit better electrical characteristics, e.g., better signal integrity, i.e., mutual coupling, voltage overshoot / undershoot, ringing, etc. This allows for better timing windows and therefore faster transmission frequencies.

[0050] As described above, the transaction layer 205, the link layer 210, and the physical layer 220 are described with respect to a specific embodiment of a PCIe protocol stack (e.g., a conventional PCIe protocol stack), but the layered protocol stack is not limited thereto. In fact, any layered protocol may be included / implemented. As an example, the port / interface may include logic implemented in hardware circuits and / or firmware to implement various layers of a layered protocol, including layers of protocols described more specifically below, such as (1) a first layer (e.g., a transaction or protocol layer) that assembles packets, a second layer (e.g., a link or data link layer) that sequences the packets, and a third layer (e.g., a physical layer) that transmits the packets.

[0051] In some implementations, the interconnect protocol may implement a cache coherent link. As an example, Ultra Path Interconnect™ (UPI™) may be utilized in high performance computing platforms such as workstations or servers, including in systems where PCIe or another interconnect protocol is typically used to connect processors, accelerators, I / O devices, etc. However, UPI is not so limited. Instead, UPI may be utilized in any of the systems or platforms described herein. Additionally, individual concepts developed may be applied to other interconnects and platforms, such as PCIe, MIPI, QPI, etc.

[0052] To support multiple devices, in one exemplary implementation, the UPI may be instruction set architecture (ISA) independent (i.e., the UPI may be implemented in multiple different devices). In another scenario, the UPI may be utilized to connect high performance I / O devices, rather than just processors or accelerators. For example, a high performance PCIe device may be coupled to the UPI through an appropriate translation bridge (i.e., UPI-to-PCIe). Also, the UPI link may be utilized in various manners (e.g., star, ring, mesh, etc.) by many UPI-based devices, such as processors. In fact, interconnect protocols such as the UPI and others described herein may be used to facilitate interconnection between dies or packages. For example, FIG. 3 illustrates exemplary implementations of various potential multi-socket configurations. As illustrated, a two-socket configuration 305 may include two links. However, in other implementations, one link may be utilized. In larger topologies, any configuration may be utilized as long as identifiers (IDs) can be assigned and some form of virtual path exists, among other additional or alternative functions. As shown, in one example, a four socket configuration 310 has links from each processor to another processor. However, in the eight socket implementation shown in configuration 315, not all sockets are directly connected to each other through their respective links. However, the configuration is supported if a virtual path or channel exists between the processors. The range of supported processors includes 2-32 in the native domain. Larger numbers of processors may be reached through the use of multiple domains or other interconnects between node controllers, among other examples.

[0053] As with other interconnect architectures, the UPI architecture includes, in some examples, the definition of a layered protocol architecture including a protocol layer (coherent, non-coherent, and optionally other memory-based protocols), a routing layer, a link layer, and a physical layer. In some implementations, each layer in the protocol stack may build and / or process information at their own level of granularity or quantum (e.g., the protocol layer in packets, the link layer in flits, and the physical layer in phits or symbols, among other examples). Note that in some embodiments, a packet may include a partial flit, a single flit, or multiple flits, based on the implementation.

[0054] In another exemplary interconnect protocol, the Compute Express Link (CXL) interconnect protocol may be utilized to provide improved high-speed CPU-to-device and CPU-to-memory interconnects designed to accelerate performance in next-generation data centers, among other applications. CXL maintains memory coherency between the CPU memory space and memory on the attached device. This allows resource sharing for higher performance, reduces software stack complexity, and lowers overall system costs, among other exemplary benefits. CXL enables communication between a host processor (e.g., CPU) and a set of workload accelerators (e.g., Graphics Processing Units (GPUs), Field Programmable Gate Array (FPGA) devices, Tensor and Vector Processor Units, Machine Learning Accelerators, Specialized Accelerator Solutions, among other examples). Indeed, accelerators are increasingly being utilized to complement CPUs in support of emerging computational applications such as artificial intelligence, machine learning, and other applications, and CXL is designed to provide a standard interface for high-speed communication.

[0055] The CXL link may be a low-latency, high-bandwidth discrete or on-package link supporting dynamic protocol multiplexing of coherency, memory access, and input / output (I / O) protocols. Among other applications, the CXL link may enable accelerators to access system memory as cache agents and / or host system memory, among other examples. CXL is a dynamic multi-protocol technology designed to support a vast spectrum of accelerators. Over a discrete or on-package link, CXL offers a rich set of protocols including I / O semantics similar to PCIe (CXL.io), cache protocol semantics (CXL.cache), and memory access semantics (CXL.mem). Based on the particular accelerator usage model, all of the CXL protocols or only a subset of these protocols may be enabled. In some implementations, CXL may be built on the well-established and widely adopted PCIe infrastructure (e.g., PCIe 5.0) and leverage the PCIe physical and electrical interface to provide advanced protocols for areas including I / O, memory protocols (e.g., allowing a host processor to share memory with accelerator devices) and coherency interfaces.

[0056] 4, a simplified block diagram 400 is shown illustrating an exemplary system utilizing a CXL link 450. For example, the link 450 may interconnect a host processor 405 (e.g., a CPU) to an accelerator device 410. In this example, the host processor 405 includes one or more processor cores (e.g., 415a-b) and one or more I / O devices (e.g., 418). A host memory (e.g., 460) may be provided with the host processor (e.g., on the same package or die). The accelerator device 410 may include accelerator logic 420 and, in some implementations, may include its own memory (e.g., accelerator memory 465). In this example, the host processor 405 may include circuitry for implementing coherence / cache logic 425 and interconnect logic (e.g., PCIe logic 430). CXL multiplexing logic (e.g., 455a-b) may also be provided to enable multiplexing of CXL protocols (e.g., I / O protocols 435a-b (e.g., CXL.io), cache protocols 440a-b (e.g., CXL.cache), and memory access protocols 445a-b (CXL.mem)) such that data of any one of the supported protocols (e.g., 435a-b, 440a-b, 445a-b) can be sent in a multiplexed manner over link 450 between host processor 405 and accelerator device 410.

[0057] In some implementations of CXL, Flex Bus™ ports may be utilized in conjunction with CXL-compliant links that flexibly adapt the device to interconnect with a wide variety of other devices (e.g., other processor devices, accelerators, switches, memory devices, etc.). Flex Bus ports are flexible high-speed ports that are statically configured to support either PCIe or CXL links (and potentially links of other protocols and architectures). Flex Bus ports allow the design to choose between providing native PCIe protocol or CXL over a high-bandwidth off-package link. The selection of the protocol applied at the port may occur during boot time via auto-negotiation and may be based on the device plugged into the slot. The Flex Bus uses PCIe circuitry to be compatible with PCIe retimers and adheres to the PCIe form factor, which is a standard for add-in cards.

[0058] FIG. 5 is a simplified block diagram illustrating an example port architecture 500 (e.g., Flex Bus) utilized to implement a CXL link. For example, the Flex Bus architecture may be organized as multiple layers for implementing multiple protocols supported by the port. For example, a port may include transaction layer logic (e.g., 505), link layer logic (e.g., 510), and physical layer logic (e.g., 515) (e.g., implemented in whole or in part in a circuit). For example, the transaction (or protocol) layer (e.g., 505) may be subdivided into transaction layer logic 525 that implements PCIe transaction layer 555 and CXL transaction layer enhancements 560 (for CXL.io) of the base PCIe transaction layer 555, and logic 530 for implementing cache (e.g., CXL.cache) and memory (e.g., CXL.mem) protocols for the CXL link. Similarly, link layer logic 535 may be provided to implement a base PCIe data link layer 565 and a CXL link layer (for CXL.io), which represents an enhanced version of PCIe data link layer 565. CXL link layer 510 may also include cache and memory link layer enhancement logic 540 (e.g., CXL.cache and CXL.mem).

[0059] Continuing with the example of FIG. 5, the CXL link layer logic 510 interfaces with a CXL arbitration / multiplexing (ARB / MUX) logic 520 that interleaves traffic from two logic streams (e.g., PCIe / CXL.io and CXL.cache / CXL.mem), among other example implementations. During link training, the transaction layer and link layer are configured to operate in either PCIe mode or CXL mode. In some examples, among other examples, a host CPU may support implementation of either PCIe mode or CXL mode, while other devices, such as accelerators, may only support CXL mode. In some implementations, a port (e.g., a Flex Bus port) may utilize a physical layer 515 based on a PCIe physical layer (e.g., PCIe electrical circuit PHY 550). For example, the Flex Bus physical layer may be implemented as a converged logical physical layer 545 that can operate in either PCIe mode or CXL mode based on the results of alternative mode negotiation during the link training process. In some implementations, the physical layer may support multiple signaling rates (e.g., 8GT / s, 16GT / s, 32GT / s, etc.) and multiple link widths (e.g., x16, x8, x4, x2, x1, etc.). In PCIe mode, the link implemented by port 500 may be fully compliant with native PCIe functionality (e.g., as defined in the PCIe specification), while in CXL mode, the link supports all functionality defined for CXL. Thus, among other examples, the Flex Bus port may provide a point-to-point interconnect capable of transmitting native PCIe protocol data or dynamic multi-protocol CXL data to provide I / O protocols, coherency protocols, and memory protocols over PCIe electrical circuitry.

[0060] The CXL I / O protocol, CXL.io, provides a non-coherent load / store interface for I / O devices. Transaction types, transaction packet formats, credit-based flow control, virtual channel management, and transaction ordering rules in CXL.io may follow all or part of the PCIe definition. The CXL cache coherency protocol, CXL.cache, defines the interaction between devices and hosts as multiple requests, each with at least one associated response message and possibly a data transfer. The interface consists of three channels in each direction: request, response, and data.

[0061] The CXL memory protocol, CXL.mem, is a transactional interface between processors and memory, using the CXL physical and link layers when communicating across the die. Among other examples, CXL.mem can be used for a number of different memory mounting options, including where the memory controller is located in the host CPU, where the memory controller is in an accelerator device, or where the memory controller is moved to a memory buffer chip. Among other example features, CXL.mem can be applied to transactions involving different memory types (e.g., volatile, persistent, etc.) and configurations (e.g., flat, hierarchical, etc.). In some implementations, the host processor's coherency engine can interface with the memory using CXL.mem requests and responses. In this configuration, the CPU coherency engine is considered a CXL.mem Master and the Mem device is considered a CXL.mem Subordinate. The CXL.mem Master is the agent responsible for sourcing CXL.mem requests (e.g., reads, writes, etc.), and the CXL.mem Subordinate is the agent responsible for responding to CXL.mem requests (e.g., data, completion, etc.). When a Subordinate is an accelerator, the CXL.mem protocol assumes the presence of a Device Coherency Engine (DCOH). This agent is assumed to be responsible for implementing coherency related functions such as snooping the device cache and updating metadata fields based on CXL.mem commands. In implementations where metadata is supported by device-attached memory, the metadata may be used by the host to implement a coarse-grained snoop filter for the CPU socket, among other example uses.

[0062] In some implementations, an interface may be provided to couple a circuit or other logic (e.g., an intellectual property (IP) block or other hardware element) implementing a link layer (e.g., 510) to a circuit or other logic (e.g., an IP block or other hardware element) implementing at least a portion of a physical layer (e.g., 515) of a protocol. For example, an interface based on the Logical PHY Interface (LPIF) specification defines a common interface between link layer controllers, modules or other logic, and modules implementing a logical physical layer ("logical PHY" or "logPHY") to facilitate interoperability, design and validation reuse between one or more link layers and physical layers for interfaces to a physical interconnect, such as in the example of FIG. 5. Also, as in the example of FIG. 5, an interface may be implemented with logic (e.g., 535, 540) to implement and support multiple protocols simultaneously. Additionally, in such implementations, a conditioning and multiplexer layer (e.g., 520) may be provided between the link layer (e.g., 510) and the physical layer (e.g., 515). In some implementations, each block of multiple protocol implementations (e.g., 515, 520, 535, 540) may interface with the other blocks via an independent LPIF interface (e.g., 580, 585, 990). If branching is supported, each branch port may have its own independent LPIF interface as well, among other examples.

[0063] While examples described herein may reference the use of an LPIF-based link layer-logical PHY interface, it should be understood that the details and principles described herein may be equally applicable to non-LPIF interfaces. Similarly, while some examples may reference the use of a common link layer logic PHY interface to couple a PHY to a controller implementing CXL or PCIe, other link layer protocols may also utilize such interfaces. Similarly, while some references are made to a Flex Bus physical layer, other physical layer logic may be employed in some implementations as well, utilizing a common link layer logic PHY interface such as that described herein, among other example variations that are within the scope of this disclosure.

[0064] Conventional die-to-die interconnects are either vendor-specific or application-specific (e.g., HBM Connect is used to connect memory-on-package). Some die-to-die interconnects (e.g., AIB from Intel, HBI and BoW from OCP Consortium) only define the physical layer and do not provide mechanisms to guarantee interoperability across dies. In fact, no current solutions exist to implement a generic die-to-die interconnect that can be used for seamless interoperability between dies and provide a slot for open innovation on a package that engineers can innovate. An improved interconnect architecture is introduced herein that implements a standardized die-to-die interface through the Universal Chiplet Interconnect Express (UCIe) protocol. UCIe not only enables a generic die-to-die interconnect solution for on-package and off-package coupling of dies, but also enables the standardization of the interface to interconnect different devices from different vendors and different factories across different technology nodes with different packaging choices for improvements on existing computing systems and implement new systems.

[0065] UCIe provides high bandwidth, low latency, power efficient, and cost effective on-package connectivity between chiplets. It addresses the projected growing demands of compute, memory, storage, and connectivity across the compute continuum and the expanding cloud, edge, enterprise, 5G, automotive, high performance computing, and handheld segments. While Moore's Law holds true, enabling increasingly compact and powerful computing blocks, the increasing on-package integration of multiple dies in mainstream commercial products such as client CPUs, server CPUs, GP-GPUs, etc. has resulted in larger die sizes to meet the increasing performance demands. However, this phenomenon has resulted in designs facing die reticle limitations. Examples include multi-core CPUs with hundreds of cores or very large fan-out switches. Even when a die can fit within the reticle limitations, multiple smaller dies connected within a package may be preferred to provide optimization along with die reuse across multiple market segments. On-package connectivity on the same die enables these scaled-up applications. On-package chiplet integration also allows designers to realize different tradeoffs in different market segments by selecting different numbers and types of die. For example, one can select different numbers of compute, memory, and I / O die according to the needs of the segment. One does not need to make different die designs for different segments, resulting in a lower SKU cost for the product.

[0066] On-package integration of chiplets allows for tailor-made solutions to be delivered in a fast and cost-effective manner. For example, different applications may require different acceleration capabilities, but with identical cores, memory, and I / O. This also allows for co-packaging of dies, where optimal processing node selection is made based on functionality. For example, memory, logic, analog, and co-packaged optics each require different processing technologies that can be packaged with chiplets. Applications that require high bandwidth, such as memory access (e.g., high bandwidth memory), are implemented as on-package integration, since the package traces are short and provide high density routing.

[0067] UCIe is an open, multi-protocol capable on-package interconnect standard for connecting multiple dies on the same package. UCIe enables the development of a vibrant ecosystem supporting disaggregated die architectures that can be interconnected using UCIe. UCIe supports multiple protocols such as PCIe, CXL, Advanced Extensible Interface™ (AXI), UPI, etc., with a raw mode available to map any selected protocol on top of a common physical and link layer (as long as both link partners support this). UCIe may also encompass elements for system-on-chip (SoC) construction such as application layers, and package-related form factors (e.g., bump locations, power delivery, thermal solutions, etc.), among other features. UCIe features operate to ensure interoperability across a wide range of devices with different performance and characteristics. Well-defined debug and compliance mechanisms are provided to ensure interoperability. UCIe may also enable support of devices in a backward compatible manner.

[0068] While UCIe supports a wide range of usage models, a subset is provided as an example illustrated herein. As mentioned above, some protocols may be explicitly mapped onto UCIe, such as PCIe and CXL. Such protocols may be mapped onto the UCIe Flit format, including raw mode. As an example, widely used protocols such as PCIe and CXL may be mapped onto UCIe for more on-package integration, such as by replacing PCIe SERDES PHY and PCIe / CXL logical PHY with link-level retries, improving power and enhancing performance, among other example features. UCIe raw mode is protocol agnostic, allowing other protocols to be mapped ad-hoc by devices, while enabling uses such as integrating standalone SERDES / transceiver tiles (e.g., Ethernet) on-package, among other example features.

[0069] FIG. 6 is a simplified block diagram 600 illustrating an example implementation of an SoC package 605 including multiple processor dies (e.g., 610, 615), accelerator dies (e.g., 620), and input / output (I / O) tiles (e.g., 625) interconnected using multiple UCIe links (e.g., 630, 635, 640, 645). In some implementations, the accelerators 620 and / or I / O tiles 625 can connect to the CPU devices (610, 615) using CXL transactions running over UCIe, thereby leveraging the I / O, coherency, and memory protocols of CXL. The I / O tiles can provide external CXL, PCIe, and DDR pins (e.g., 650, 655) of the package 605. The accelerators may connect to the CPUs using PCIe transactions running over UCIe. On-package CPU-to-CPU connectivity may use a UCIe interconnect running a coherency protocol, among other examples. In some implementations, all components on the exemplary SoC package may be interconnected using UCIe links. In other examples, one or more blocks (e.g., memory blocks 660, 665, 675) may be connected using other protocols or links (e.g., DDR, etc.), among other exemplary implementations. Additionally, as shown in FIG. 6, a UCIe link (e.g., 685) may be utilized to couple to an on-package retimer device 670. The retimer 670 may be utilized to extend the physical reach of the UCIe link (e.g., 680) and enable off-package connections (e.g., to other components on other packages), among other exemplary components.

[0070] A variety of packages may support UCIe links. For example, among other examples, standard two-dimensional (2D) packages may be utilized, along with 2.5D and 3D advanced packaging options. FIG. 7 is a simplified block diagram illustrating an exemplary application using a standard 2D package. The 2D package may be used for low-cost, long-reach (e.g., 10 mm to 25 mm) interconnects using traces 705, 708 on an organic package / substrate 710, further providing significantly better BER performance compared to off-package SERDES. As shown in FIG. 7, a first die 715 may be coupled to a second die 720 and a third die 725 by respective links implemented using traces (e.g., 705, 708) on the substrate 710.

[0071] 8A-8D are simplified block diagrams 800a-d illustrating example applications using UCIe Advanced (e.g., 2D) options. Such packaging technology may be used for performance optimized applications. As a result, the channel reach is short (e.g., less than 2 mm) and the interconnect is optimized for high bandwidth and low latency with best performance and power efficiency characteristics. In one example shown in FIG. 8A, the links may be implemented on silicon bridges 805, 810 (e.g., Embedded Multi-Die Interconnect Bridge (EMIB)). In the example of FIG. 8B, an interposer 815 on package 710 and UCIe links (e.g., 820, 825) may be provided on the interposer 815 (e.g., in a chip-on-wafer-on-substrate (CoWoS) application). In yet another example of an advanced package shown in FIG. 8C, silicon bridges (e.g., 830, 835) may be provided on a fan-out organic interposer 840 provided on the package 710, and UCIe links may be implemented on the silicon bridges 830, 835. FIG. 8D illustrates another example where a combination of standard and advanced options is employed. For example, a base die 850 may be coupled to a companion die 855 in a 2D manner by a first link utilizing standard package traces 860, and the base die 850 is coupled to another base die 865 by a bridge-based link 870. In another example, the base die 865 couples to a vertical chiplet 875 via a 3D UCIe link, among other example architecture and package applications.

[0072] UCIe supports two broad usage models. The first usage model, as introduced above, involves package-level integration to provide power-efficient and cost-effective performance. Accordingly, board-level mounted components such as memory, accelerators, network devices, modems, etc. may be integrated at the package level, even on the same package, with applicability from handheld to high-end servers with dies from multiple sources connected through different packaging options. The second usage is to provide off-package connectivity using different types of media (e.g., optical, electrical cable, mmWave) with UCIe retimers, transport underlying protocols (e.g., PCIe, CXL) at the rack or even at the pod level, deriving better power efficiency and cost-effective performance at the edge and data center with load-store semantics from the node level to beyond the rack / pod level, enabling resource pooling, resource sharing, and even message passing. 9 is a simplified block diagram illustrating an example rack server system including server-scale interconnects between devices (e.g., UCIe may be utilized to facilitate off-package connectivity). In such implementations, retimers may be provided (e.g., on-package top of rack servers (TORS)) to facilitate off-package connectivity. For example, off-package UCIe links may be utilized to couple a device (e.g., processor device 905) on a first chassis 910 to another device (e.g., processor 915) on another second chassis 920 (e.g., via switch device 925), among other example implementations.

[0073] UCIe is a layered protocol, as shown in block diagram 1000 of FIG. 10. Physical layer 1005 is responsible for electrical signaling, clock, link training, and sideband signaling. Die-to-Die (D2D) adapter block 1010 provides chiplet link state management and parameter negotiation. It ensures reliable delivery of data, optionally through cyclic redundancy check (CRC) and link-level retry mechanisms. If multiple protocols are supported, the D2D adapter defines the underlying coordination framework for inter-protocol data coordination on the link. In one exemplary implementation, UCIe may provide 256-byte Flits (or flow control units) and define the underlying transfer mechanism if the adapter is responsible for reliable transfers. As mentioned above, one or more other interconnect protocols may be natively mapped to the UCIe Flit format. In one example, UCIe ensures seamless interoperability by mapping PCIe and CXL protocols and leveraging the existing ecosystem of these protocols. For example, PCIe and CXL allow already deployed SOC construction, link management, and security solutions to be leveraged for UCIe. The usage models addressed are also comprehensive. Among other example benefits and use cases, data transfer with direct memory access, software discovery, error handling, etc. are addressed by PCIe / CXL.io, memory use cases are handled through CXL.Mem, and cache requirements for applications such as accelerators are addressed by CXL.cache. UCIe also defines a "streaming protocol" that can be used to map any other protocol to the Flit format defined in UCIe.

[0074] The protocol layer 1015 may follow any interconnect protocol, including, in some cases, protocols natively mapped to UCIe (e.g., CXL, PCIe, etc.), other well-defined or standardized interconnect protocols that may utilize UCIe's streaming or raw modes, or proprietary or vendor-defined protocols that may also utilize UCIe's streaming or raw modes. The protocol layer may implement the transaction layer of the corresponding protocol and implement flow control schemes defined in the protocol, among other protocol-specific functions. A logical block implementing the protocol layer 1015 (e.g., in software, firmware, and / or hardware) may interface the UCIe D2D adapter 1010 with a Flit-aware D2D interface (or FDI) 1020. The FDI interface may be defined in the UCIe protocol specification and may define a set of signals for communication and negotiation between the protocol layer block 1015 and the D2D adapter block 1010. For example, signals or messages may be defined in the FDI 1020 by which the D2D adapter 1010 informs the protocol layer (e.g., based on negotiation with a link partner device (e.g., a D2D adapter of the link partner device)) of the Flit format to be utilized (e.g., mapping Flit format, streaming mode Flit format, raw mode Flit format, etc.) which the protocol layer 1015 may use to adapt the data of the corresponding protocol to the identified Flit format, among other examples.

[0075] Through the provision of streaming and raw modes, UCIe allows the option to transport data of any protocol over the UCIe link. Protocol data can be transported in Flit format or as raw fabric wires if necessary. Some protocols may be explicitly mapped, allowing on-board components such as accelerators, memory expanders, and I / O expanders to work seamlessly on-package. Flit mode includes an operational mode in which error detection and / or error correction bits / bytes (e.g., CRC) are inserted into the defined Flit format by the D2D adapter. Flit mode includes an operational mode in which a Flit of another underlying protocol of data (e.g., PCIe, UPI, CXL, GenZ, AXI, etc.) is mapped into the UCIe Flit format along with streaming mode (possibly where data of any other protocol is encoded or entered into the Flit format (e.g., including CRC and any other specified fields defined in the UCIe Flit)). Raw mode, on the other hand, is a mode of operation where all bytes of the UCIe Flit are input by the protocol layer (and the D2D adapter accepts the Flit as is, without adding CRC, retries, or other values ​​to the Flit, and forwards the data without any modification from the protocol layer to the physical layer). In some examples, raw mode CRC, retries, FEC, etc. may be handled entirely by the protocol layer (rather than the UCIe D2D adapter), among other examples.

[0076] In some implementations, the D2D adapter 1010 may be provided as a separate block of circuitry (e.g., from those implementing the protocol layer block and / or the physical layer block 1005). In other implementations, one or both of the logic implementing the protocol layer or the physical layer may be collocated or integrated with the logic implementing the D2D adapter block 1010 (e.g., hardware circuitry), among other example implementations. The D2D adapter 1010 coordinates the protocol layer 1015 (e.g., via the FDI 1020) and the physical layer 1005 (e.g., via the raw D2D interface (RDI) 1025) to ensure successful data transfer across the UCIe link. Providing such standardized interfaces (e.g., FDI and RDI) for coupling blocks (e.g., IP blocks) implementing each layer of the protocol allows vendors and SoC builders to easily mix and match different layers from different IP providers with lower integration costs and faster time to market. (For example, enabling protocol layer blocks to work with D2D adapters and physical layer blocks from different vendors that conform to the interface handshakes provided in this specification.) Furthermore, where interoperability testing in post-silicon has greater overhead and costs associated with it, consistent understanding and development of bus functional models (BFMs) based on such interfaces may enable easier IP development for such UCIe protocol stack implementations, among other exemplary benefits.

[0077] The D2D adapter 1010 helps facilitate minimization of logic in the main data path and realizes a highly latency-optimized data path for the protocol Flit. The D2D adapter may include logic to implement various functions that may be utilized by various protocols (of the protocol layer 1015), such as reconciliation and multiplexing functions, error detection and / or error correction codes (e.g., Cyclic Redundancy Check (CRC)), retry schemes, among other example functions. As an example, a protocol layer implementing CXL may be coupled to the D2D adapter. When transporting the CXL protocol, the ARB / MUX functionality of the D2D adapter 1010 may be utilized to perform reconciliation / multiplexing of multiple sub-protocols of CXL (e.g., CXL.mem, CXL.io, CXL.cache). Error detection, error correction, retry, and other functions provided by the D2D adapter 1010 may be utilized to support a target bit error rate (BER) requirement (e.g., based on the speed to be implemented on the link). For example, if the raw BER is less than 1e-27, a CRC and retry scheme may be provided by the D2D adapter 1010. The D2D adapter 1010 may also be responsible for coordinating higher level link state machines and link establishment protocols, protocol options for parameter exchange with a remote link partner, and power management coordination with the remote link partner, if supported. For example, two link partner devices coupled with a UCIe link may utilize their respective D2D adapters to perform, among other examples, training of the link (e.g., through progression through various link states defined in a link state machine) and determination of parameters to be used during operation of the link.

[0078] A communication port of a computing device may be constructed from a protocol logic block, a D2D adapter block, and a physical layer block. The protocol logic block may interface with the D2D adapter block via an FDI interface, and the D2D adapter may interface with the physical layer block using an RDI interface. Each of the FDI and RDI interfaces may be implemented with physical pins, traces, conductors, or other communication paths along which the respective signals of the defined interfaces may be communicated. The protocol logic block (e.g., 1015), the D2D adapter block (e.g., 1010), and the physical layer block (e.g., 1005) implement the respective layers of the UCIe protocol, enabling these devices to establish a UCIe-compliant link and communicate with another link partner computing device using the UCIe protocol.

[0079] The physical layer of the UCIe stack may be implemented through multiple physical layer modules to form the physical layer of the corresponding link. The modules form the main data path (e.g., on the physical bumps or pins of a port) and are organized as groups of lanes. The modules form the atomic granularity of the structural design implementation of the UCIe analog front end (AFE). The number of lanes per module may be based on whether a standard or advanced package is utilized in the implementation. A single D2D adapter may be coupled to a physical layer implemented as one or more modules (for bandwidth scaling). The UCIe physical link (and the respective modules forming the physical layer) may include a sideband connection (formed of a first subset of lanes) and a main band connection (formed of a second subset of the physical layer lanes). The UCIe module / physical layer main band connection provides the main data path of the UCIe. The main band connection may include a forward clock, a data valid pin, and N data lanes per module. The sideband connection is used for parameter exchange, register access for debug / compliance, and coordination with the remote partner for link training and management. The sideband connection may include a forward clock pin and a data pin in each direction. In one exemplary implementation, the forward clock may be fixed at a frequency of 800 MHz regardless of the speed of the main data path (e.g., at a different frequency than the main data path (main band) speed). In some implementations, the sideband logic for the UCIe physical layer may be powered using auxiliary power, placing the sideband logic in an "always on" domain. Each module may include its own respective set of sideband pins / bumps / lanes.

[0080] As introduced above, the retimer may be used to extend UCIe connectivity beyond the package using off-package links (e.g., electrical or optical cable communication media), for example, to connect packages at the rack / pod / blade level. The UCIe retimer implements a UCIe interface to the die that connects to its local package according to the UCIe protocol and ensures that Flit is provided across two dies on different packages. UCIe may be used to transport any protocol, including load-store protocols such as PCIe or CXL, over a long reach alternative medium such as optical. Referring to FIG. 11, a simplified block diagram 1100 is shown illustrating an example system including two packages (e.g., 1110, 1115) connected by a UCIe link 680. As introduced above, the UCIe retimer may be used to enable different types of off-package interconnects 680 to extend the channel reach between two UCIe dies on different packages (e.g., 1110, 1115). A respective UCIe retimer (e.g., 670, 1125) may be provided in each package to enable an off-package UCIe link (e.g., 680). Additionally, the UCIe retimer 670, 1125 of each package may be coupled to one or more of the dies (e.g., 1130, 1135) on that package by a respective UCIe link (e.g., 685, 1145). The dies (e.g., 1130, 1135) may transfer data to the corresponding retimer (e.g., 670, 1125) for communication to other dies on other packages via the off-package link 680 facilitated through the retimer. With respect to the on-package link (e.g., 685) coupling the retimer (e.g., 670) to the on-package die (e.g., 1130), the off-package link may be much longer in physical length. An off-package retimer (eg, 1125) to which a retimer (eg, 670) is connected using an off-package link 680 may be referred to as a "remote retimer partner."Data generated by a die (e.g., 1130, 1135) (e.g., contained in a Flit of a particular format) may be transferred by a retimer (e.g., 670, 1125) to other retimers (e.g., 1125, 670) via an off-package link 680.

[0081] The role of each UCIe retimer (e.g., 670, 1125) in an exemplary system may include reliable transport of Flits via an off-package interconnect 680. In one example, the retimer (e.g., 670, 1125) is configured to implement and support the use of forward error correction (FEC) and error detection (e.g., cyclic redundancy check (CRC)) natively defined by the underlying specification of the protocol it transports, so long as the external interconnect complies with the underlying error model (e.g., BER and error correlation) of the specification corresponding to the protocol (e.g., PCIe or CXL) it transports. In such an example, the UCIe link may be configured to utilize raw mode to tunnel the native bits of the protocol (e.g., PCIe or CXL Flit) it transports. In this scenario, the queue size (e.g., protocol layer buffers) is adjusted on the UCIe die to match the underlying round-trip latency. In another example, the retimer may be configured to support and provide defined FEC, CRC, and retry functions that handle the BER of the off-package interconnect. In this case, Flit receives three independent links, and each UCIe retimer performs an independent Ack / Nak for retries with its in-package UCIe die and a separate independent Ack / Nak for retries with the remote retimer partner. As another example, the retimer may provide its own FEC by replacing the native protocol's (e.g., PCIe or CXL) defined FEC with itself, or by adding its FEC in addition to the native protocol's defined FEC, but taking advantage of the built-in CRC and replay mechanisms of the underlying protocol. In this example, queue sizes (e.g., protocol layer buffers, retry buffers, etc.) must be adjusted on the UCIe die to match the underlying round-trip latency.

[0082] The UCIe retimer may be responsible for resolution of link layer and protocol layer parameters with the remote retimer partner to ensure end-to-end (E2E) interoperability between the UCIe dies. For example, the retimer (e.g., 670, 1125) is permitted to enforce the same link width, speed, protocol (including any associated protocol-specific parameters), and Flit format on both packages (e.g., 1110, 1115). The specific mechanisms of resolution, including message forwarding for parameter exchange across the off-package interconnect, may be retimer implementation specific, and they operate to ensure a consistent mode of operation, taking into account their own capabilities as well as the UCIe dies (e.g., 1130, 1135) capabilities on both packages (e.g., 1110, 1115). However, for robustness of the UCIe link (e.g., 685, 1145) and to avoid unnecessary timeouts in case the external interconnect 680 with the remote retimer partner needs more time for link establishment or parameter resolution, a "stall" response may be defined for the associated sideband message that may possibly be delayed. The retimer may respond with a "stall" response while waiting for or negotiating with its remote retimer partner (e.g., according to specification-defined rules) to avoid such unnecessary timeouts. It is the retimer's responsibility to ensure that the UCIe link is not stalled indefinitely. As another function, the UCIe retimer (e.g., 670, 1125) with its remote retimer partner may be responsible for managing the link state or RDI state defined in the D2D adapter (e.g., adapter link state machine (LSM)) to ensure correct E2E operation, among other example considerations.

[0083] The retimer may also provide flow control and backpressure management logic to manage data sent from its on-package UCIe die (e.g., 1130, 1135). For example, data sent from the UCIe die to the UCIe retimer may be flow controlled using credits. The credits used to manage this link (e.g., 685, 1145) may be in addition to any credit protocol defined for the underlying protocol of the data being sent (e.g., PRH, PRD credits in PCIe). UCIe die-to-die (D2D) credits may be defined for flow control across the two UCIe retimers, and any data sent to the UCIe retimer is gradually consumed by the remote UCIe die without any other dependencies. The UCIe retimer (e.g., 670, 1125) may implement receiver (Rx) buffers (e.g., 1150, 1155) for Flits it receives from the UCIe die (e.g., 1130, 1135) within its package. The receiver buffer credits are provided to the UCIe die during the initial parameter exchange for the D2D adapter, and the UCIe die may not need to send any data to the UCIe retimer if it does not have credits to send. In one exemplary implementation, one credit corresponds to 256B of data (including any FEC, CRC, etc.). The return of the credits may be overloaded or encoded on / within the valid frame signal in one exemplary implementation. The credit counter of the UCIe die may be reassigned to the initial provided value whenever the RDI state transitions away from active. The UCIe retimer may drain or discard data in its receiver buffer (if applicable) before re-entering the active state. In some examples, data sent from the UCIe retimer to the UCIe die may not be flow controlled at the D2D adapter level. The UCIe retimer may instead have its independent flow control, along with other UCIe retimers if necessary, in other exemplary implementations.

[0084] 12 is a simplified block diagram 1200 illustrating an example UCIe retimer 670. Generally, the UCIe retimer 670 may enable adoption of the UCIe protocol on larger scale applications and systems and allow different transmission mediums and technologies (e.g., optical, mmWave, or electrical cable) to be utilized to physically connect different packages, including packages within applications such as automotive, Internet of Things (IoT) applications, industrial applications, rack / pod server split applications, scale-out systems, among others, while still maintaining high bandwidth beyond those technologies (e.g., electrical point-to-point interconnects with defined lane assignments and signaling characteristics) defined for use with the UCIe protocol. In the example shown in FIG. 12, the exemplary UCIe retimer 670 may be provided with two or more ports, including at least one local UCIe port (e.g., 1205) that couples to other components (e.g., dies) on the same package via respective UCIe links (e.g., 685), and one off-package, or external port (e.g., 1210) that implements an off-package interconnect (e.g., 680). The external port 1210 may be adapted to implement and communicate via a particular technology and corresponding protocol, such as a technology that enables higher speed, longer reach channels, such as optical or mmWave, among other examples. The external port 1210 may connect to an external port of another partner retimer device (e.g., on a different package) via an off-package link 680. This partner timer may further be coupled to one or more dies on that package, thereby facilitating die-to-die communication between packages via the off-package link 680, and supporting retimers (e.g., 670) such as those introduced in the example of FIG. 11.

[0085] Each port (e.g., 1205, 1210) of a device including a retimer device (e.g., 670) may include circuits, modules, or other logic that implement one or more layers of a UCIe protocol such as those outlined in the example of FIG. 10. For example, in one example, the local UCIe port 1205 may implement the D2D adapter 1010 and the UCIe PHY 1005. In some examples, the retimer 670 may be completely protocol agnostic (e.g., to allow the retimer to be flexibly deployed in various applications). In other examples, the retimer's port may further implement one or more protocol layers of an underlying protocol to be supported by the system, among other example implementations. An external port (e.g., 1250) may include an implementation of a D2D adapter (e.g., 1250). The external port of the retimer's D2D adapter may be specialized or may differ from a typical UCIe D2D adapter in its additional functionality and may be employed to facilitate an off-package link (e.g., 680) that enables associated logic and circuitry to process data passed to it over a UCIe link (e.g., 685) from a die coupled to the retimer (e.g., 680) from a standpoint of both speed and reliability. Such additional functionality may include, for example, enhanced or specialized error detection and / or correction functionality (e.g., capable of bringing the off-package interconnect medium within an acceptable or compatible BER range), Flit packing logic (to adapt data received on the UCIe link 685 for transmission over the different technology of the off-package link 680), and flow control functionality (e.g., implemented using flow control circuitry 1255, among other example functions and logic).

[0086] 12, the retimer 670 may be capable of passing data received from one port (e.g., 1205, 1210) at a rate that allows the UCIe link to continue to function in a Flit-based mode of operation (e.g., ports continuously sending and receiving data in fixed-sized chunks, or in a stream of Flits) without data loss and forwarding the same data on the other port (e.g., 1210, 1205). Accordingly, the retimer logic 1215 may be implemented in hardware circuitry and / or firmware of the retimer 670 to facilitate retiming of data in both directions between the ports 1205, 1210. If the retimer 670 may utilize a different transmission medium and technology defined or envisioned for the die-to-die UCIe link (e.g., 685) through its external off-package link (e.g., 680), the retimer logic 1215 may include a channel adapter circuit 1225 to modulate, transform, or otherwise adapt the data from the UCIe medium (e.g., a multi-lane electrical channel) to one or more different channel types (e.g., optical, mmWave, electrical channels (e.g., Ethernet) having different widths or characteristics, etc.). The retimer 670 may include one or more configuration registers 1220 to configure the performance of the retimer 670. Such registers (e.g., 1220) may be written and / or read by system software and may be entered or defined according to the UCIe protocol during training of the link to identify, among other example information, the characteristics and configuration of the parameters of the retimer. The register values ​​may be utilized by a controller (e.g., 1230) implemented in the retimer 670 to drive how the retimer and its ports (e.g., 1205, 1210) operate based on this training and configuration, among other example uses.

[0087] The retimer 670 may implement one or more instances of a UCIe PHY (e.g., 1005) on one or more of its ports (e.g., 1205). FIG. 13 is a simplified block diagram 1300 illustrating an example physical layer module of a UCIe device. The physical layer may include a logical physical layer (logical PHY) 1305, or PHY logic, a number (M) of sideband lanes 1310 and a number (N) of main band lanes 1315. As described above, the main band lanes include one or more forward clock lanes 1320, a number of data lanes 1325, and a valid lane 1330. The logical PHY 1305 may perform link training, lane repair, lane inversion, scrambling / descrambling, sideband training and forwarding, among other functions for the link. The physical layer may couple to a D2D adapter (e.g., implemented on a separate hardware or IP block) via an RDI interface 1025. Main function data may be communicated using the main band lanes 1315, while training and configuration data (e.g., for use in reading and writing data from the retimer registers) may utilize side band lanes (e.g., 1310). As introduced above, the physical layer for a UCIe link may be implemented using one or more PHY modules based on the specified width of the link to be implemented.

[0088] As introduced above, the D2D adapter of the retimer may implement error detection and correction functions for the external off-package link implemented with the retimer. In some examples, the D2D adapter may include and utilize standard UCIe error detection and correction functions. In other examples, depending on the nature of the off-package communication channel, the speed of the link, the application, or other factors, the retimer D2D adapter may alternatively or additionally implement other enhanced error detection and correction functions to achieve a comparable level of reliability on the external link. As an example, the D2D adapter of the retimer may support the same FEC / CRC natively available in a standard UCIe D2D adapter (e.g., based on the FEC / CRC outlined in the PCIe 6.0 specification) and may utilize this error detection and correction functionality and scheme as long as the external interconnect complies with the corresponding bit error rate (BER) threshold (e.g., PCIe 6.0 BER rules) across the two UCIe dies (e.g., 10-6 low BER, Flit error rate ≦3×10-5). If achieving this reliability threshold is not possible using the native error detection and correction (e.g., FEC / CRC) functionality, the retimer may instead or additionally utilize other FEC, CRC, retries, or other techniques that achieve the required BER characteristics of the communication channel. In a die-to-die combination with a retimer and an off-package link, the Flit receives three independent links: a UCIe link from the first die to the first retimer, an off-package link from the first retimer to the second retimer, and a second UCIe link from the second retimer to the second die. Accordingly, each UCIe retimer may perform its own independent ack / nak with the UCIe die to which it is connected, and a separate independent Ack / Nak with the other UCIe retimers. The UCIe retimers, in one example, may utilize different FECs (e.g., adapted to the technology of the outer channel) to meet a defined Flit error rate.For example, the retimer may add its own FEC, either by removing the native FEC (e.g., PCIe / CXL FEC) or by adding its own FEC on top of the PCIe / CXL FEC, while taking advantage of the built-in CRC / replay mechanism of the underlying protocol (e.g., PCIe / CXL), among other examples. In such an example, the queue size of the retimer receive buffer may be adapted to match the underlying round-trip latency, among other exemplary configurations.

[0089] Data transmitted over the off-package link maintains the per-Flit transmission characteristics. Indeed, as described above, the retimer may reuse CRC, FEC, and other native UCIe features. The associated fields may be the defined UCIe Flit format. In other applications and modes (e.g., when a retimer-specific CRC / FEC is used), UCIe low mode may be utilized instead in the off-package link. FIGS. 14A-14C show example Flit formats that may be supported by a protocol stack implemented on the external port of the UCIe retimer. For example, FIG. 14A shows an example low mode format defined in UCIe. As described above, low mode may effectively allow data of any type, protocol, or format to be transmitted over the UCIe link. To the extent that CRC, FEC, retries, or any other features are employed in low mode, the retimer may implement additional logic (e.g., in the retimer logic and / or external port circuitry) to implement these independently of the D2D adapter. In the example of FIG. 14A, a standard raw mode format 1400a is shown in which all 64 bytes (1402) of the transmission medium are defined by the protocol layer. In an advanced package configuration, the longer raw mode data format may be provided with additional bytes provided through redundant or spare lanes in the advanced package configuration. These spare lanes may be utilized to transmit parity data or other information. In one example, the additional parity information transmitted on the spare lanes may include 1 B of parity for every 8 UI of data transferred across the physical link.If parity is not possible or if spare lanes are not available to transmit parity, the D2D adapter can check and count CRC errors used by the underlying protocol to aid in characterizing the D2D link (particularly useful for characterizing or debugging links where the optical connection is the dominant source of errors), among other example functions and features, which may be added by the D2D adapter even when the link is operating in raw mode. When operating in raw mode (as negotiated by the D2D adapter), the Flit data is mapped as is onto the UCIe wire, byte by byte (e.g., 64B chunks shifted over time).

[0090] FIG. 14B illustrates an example Flit mapping representation 1400b for an example Advanced Package use case where one or more spare lanes (e.g., 1405, 1410) are available. For example, in the example of FIG. 14A, an optimized mode for 256B optimized UCIe Flit mapping is shown. Here, one spare lane 1410 is used by the D2D adapter to transmit CRC and another spare lane 1405 is used to transmit Flit header and credit return information. The protocol layer may present information in 256B Flit. Through this optimization, enabled by the spare lanes 1405, 1410, only TLP information (e.g., 1415, 1420, 1425, 1430) is transmitted on the main band data path 1440. In some implementations, the retimer may wrap its special FEC, CRC, and / or replay data as necessary. FIG. 14C shows an example implementation of native 256B Flit used in PCIe6.0 / CXL3.0 mapping. In this example, 2B of the first 64B of data are reserved for information input from the D2D adapter, 240B of the next Flit data (1440, 1445, 1450, 1455) are provided by the protocol layer, and then another 14B (1460) is reserved for the D2D adapter. The UCIe retimer may support these example formats, among others, as defined in the UCIe specification. In the mandatory mechanism, the native 256B Flit used in PCIe6.0 / CXL3.0 is used. This format may be used, for example, for PCIe, CXL, and streaming protocols. The UCIe retimer may replace the FEC with its own FEC in some implementations and may use an 8B CRC and replay mechanism of the underlying PCIe 6.0 / CXL 3.0 protocol across two UCIe dies in other example implementations. Table 1 shows an example of how various underlying protocols may be mapped to UCIe retimer modes in other possible example implementations. [Table 1] Table 1: Exemplary protocol-retimer mode mapping in a system

[0091] The technology employed in the off-package communication channel supported by the retimer may support data rates or frequencies lower than, equal to, or higher than those employed in a standard UCIe link channel (e.g., as defined in the UCIe PHY). To facilitate die-to-die bonding over an off-package link, the die partners negotiate a specific bandwidth and frequency and expect the other die partner to match these same characteristics. Accordingly, the retimer is tasked with maintaining this agreed-upon frequency in their management of the off-package link (e.g., during training of the link) to preserve correct functioning throughout the bonding of the two dies. In some examples, the die may transmit data more reliably or at a faster rate than the retimer can transfer data over the off-package link. In response, the retimers implement flow control schemes, respectively, to further aid in reliable data transmission while preserving the flit-based encoding of the link. For example, a flit-level flow control mechanism may be implemented for flits transmitted from the die to the UCIe retimer by the unit that is the flit. The UCIe retimer may throttle the UCIe die as necessary (e.g., to provide cycles for the retimer to perform its own link management tasks (e.g., FEC, frequency conversion, sideband messaging, etc.)).

[0092] In one example, each retimer may implement a credit-based flow control scheme for Flits sent from the die in the UCIe retimer direction (e.g., based on the retimer's Rx FIFO buffer). In one example, each credit may correspond to a single (e.g., 256B) Flit. The UCIe retimer may utilize a FIFO from which it provides credits to its local UCIe die. This may be tracked using a credit counter in the die's D2D adapter (to correspond to the retimer die's buffer) and is applied on top of other protocol-level credits that may be implemented based on the underlying protocol of the die's protocol layer. The UCIe retimer may throttle the UCIe die through the utilization of these credits. In one example implementation, a credit check may be required to send any Flit between the die and the retimer (e.g., even a null Flit may not be sent if the die is out of credits). Other throttling mechanisms may also or alternatively be employed, including the use of null Flits or other Flits designed to backpressure the link at a per-Flit level (e.g., when credits are not currently available).

[0093] 15, a signal diagram 1500 is shown illustrating an exemplary scheme for releasing retimer credits used for flow control in a UCIe retimer, such as the one introduced above. In one exemplary implementation, the release (or return) of credits may be overlaid on a valid signal provided to the main band portion of the UCIe PHY. For example, a valid signal asserted on the valid pin 1330 may indicate that the next 64 bytes received on the data pin 1315 constitute valid data. In one example, rather than asserting 8 UI of valid signals corresponding to the 64 bytes, the valid signal for the 64 bytes may be asserted for only the first 4 UI to indicate the validity of the data on the data lane 1315. Accordingly, the remaining 4 UI of the valid signal may be utilized for the release of credits. As one exemplary implementation, an encoding may be defined for the valid signal that provides guaranteed triple bit flip detection and enables the encoding of the release of credits, as follows: 1111_1111: Flit data transfer valid + 1 credit released (for example, valid signal 1505) 1111_0000: Flit data transfer enabled + no credits released (e.g. enabled signals 1510, 1515, 1520) · 0000_1111: Flit data transfer no +1 credit release 0000_0000: No Flit data transfer + no credits released

[0094] In some examples, any error may trigger a retrain of the link, the local UCIe die loses all Flit credits released to it, and the UCIe retimer then re-releases the credits for the remaining FIFO entries. In some implementations, there may be no explicit local flow control implementation from the local UCIe retimer to the UCIe die at the D2D adapter level. Any error in the valid signal may cause the link to recover and retrain. The UCIe retimer releases only credits for the space available in its FIFO buffer. It must be guaranteed that all Flits recognized locally by the UCIe retimer remain in the retimer FIFO and are sent by the retimer via an off-package link to the UCIe retimer on the other package. In some examples, among other example implementations, rather than requiring an explicit credit return at the Flit or transaction level, the credit counters may be reinitialized to the values ​​they were originally provided.

[0095] 16A, a simplified block diagram 1600a of an exemplary system is shown that includes a first UCIe die 1130 connected to a first UCIe retimer 670 by a first UCIe link 685. The first UCIe retimer 670 is connected to an off-package second UCIe retimer 1125 by an off-package external link 680, which is connected to a second UCIe die 1135 via a second UCIe link 1145. The die's UCIe ports (e.g., 1605, 1620) and the retimer's local ports (e.g., 1205, 1615) may implement UCIe PHYs on the local links 685, 1185, respectively, that include main band lanes (e.g., 1625, 1635) and side band lanes (e.g., 1630, 1640). The D2D adapters of the dies 1130, 1135 may communicate with each other using sideband channels 1630, 1640, which are carried by the retimers 670, 1125 over the external link 680 to negotiate link characteristics and perform training of the three links 685, 680, 1145 that form the communication channels between the dies 1130, 1135.

[0096] 16B shows a flow diagram illustrating an example flow for establishing communication channels and composite links 680, 685, 1145. A local UCIe link (e.g., 685, 1145) coupling a local die to a local retimer may be established through initialization 1650a-b of a sideband communication channel of the respective link 685, 1145. Upon establishment of the sideband channel, the local die and the retimer may utilize the sideband channel and perform initialization 1654a-b (e.g., to establish parameters of the local link (e.g., 685, 1145)) and training 1656a-b of the local link (e.g., to establish data rates of the local link). An external link 680 coupling the retimer may be established 1652 independently (e.g., before, after, or in parallel) of the local link 685, 1145. Once established (at 1658), the external link 680 may be utilized to enable inter-retimer communication, which in turn allows the off-package dies to communicate with each other and determine appropriate end-to-end parameters to be applied across the communication channels coupling the dies via the external link 680. Accordingly, in one implementation, the D2D adapters of the remote dies may exchange messages 1662 to communicate and negotiate their respective remote retimer parameters 1660a-b. During these message exchanges (e.g., 1662, 1666, etc.), the role of the retimer is to negotiate state transitions with its remote retimer partner and ensure that the different UCIe dies are synchronized and do not time out waiting for responses. As an example, if UCIe die 0 sends an active request message to the adapter LSM to UCIe retimer 0, among other examples, UCIe retimer 0 must resolve with UCIe retimer 1 that the active request message has been forwarded to UCIe die 1, which responds with an active status message before responding to UCIe die 0 with an active status message.Along the same lines, the external off-package interconnects will not be transitioned to a low power state unless all associated states on UCIe die 0 and UCIe die 1 have reached the corresponding low power state.

[0097] With these established 1660a-b parameters, each die may further train (or retrain) its main band channel 1664a-b based on these negotiated retimer parameters. Additional messages 1666 may be communicated between the die's remote D2D adapters to establish and match (at 1668a-b) the overall speed of the channel. If there is a data rate difference between the two local links 685, 1145, each local UCIe link 685, 1145 may be retrained to achieve a speed (data rate) match. If a speed (data rate) match is achieved, each UCIe link (e.g., 685, 1145) proceeds to active through protocol level link initialization (LINK INIT) 1670a-b to place these links 685, 1145 in an active operating state. With all three links 680, 685, 1145 active and trained, function data 1674 may then be communicated between the two remote dies using their respective retimers (and the external link 680 coupling their respective packages), among other exemplary implementations.

[0098] 17 is a simplified flow diagram 1700 illustrating an example state machine for initializing a flow in a communication channel including an off-package external link facilitated by two UCIe retimer devices. A reset state 1705 may be triggered at boot, startup, or in response to a training error (e.g., 1708) to cause the link to retrain, beginning with sideband initialization 1710, followed by main band initialization 1715, and main band training 1720. Depending on the parameters of the UCIe link established, a protocol level link initialization 1725 may proceed to bring the link to an active state 1730. One or more low power states 1732 (e.g., lower power than the active state 1730), such as L1 and L2 low power states, may be provided and entered from the active state 1730. The L1 state may exit into the main band training state 1720, and the L2 state may exit into the reset state 1705. If an error or failure is detected during operation, among other example implementations of a UCIe link state machine, the retrain state 1735 may be entered, forcing a retrain of the main band (eg, at 1720).

[0099] As mentioned above, one or more sets of registers may be associated with (and hosted on) UCIe retimer devices to aid in the configuration of communication channels established with these retimer devices. The UCIe specification may define a number of registers used in establishing and operating a UCIe link. As an example, UCIe may define one or more Designated Vendor Specific Extensions (DVSEC) registers. The UCIe link DVSEC registers may be provided to serve as a register set of basic functions used in operating a UCIe link. Software may use the presence of the DVSEC registers to distinguish between UCIe devices and standard PCIe or CXL devices. The UCIe retimer may implement a UCIe link DVSEC register 1800a such as that depicted in FIG. 18A. The UCIe link capability registers 1805, control registers 1810, and status registers 1815 may contain the D2D / PHY capabilities / control / status of the retimer. The register locator register 1820 may provide up to three locations for adding additional registers, such as remote agent provided link / protocol features and any retimer specific registers not defined in the UCIe specification, among other examples. For example, FIG. 18B illustrates an example retimer D2D / PHY register block that can be placed in any of the three register locator areas. FIG. 18C illustrates an example retimer test / compliance register block that can also be placed in any of the three register locator areas. FIG. 18D illustrates an example retimer implementation specific register block that may be placed in any of the three register locator areas.

[0100] The UCIe retimer registers may be accessed (e.g., by software) using sideband messaging on a sideband lane of the local UCIe link. In some implementations, a window register may be provided for software to interact with the retimer registers. For example, the UCIe retimer registers may be directly accessible from host software in some implementations. Alternatively, software may access the retimer registers via a window register using a sideband interface. For example, a window register may be available to access the UCIe retimer registers on the retimer closest to the host via a UCIe link DVSEC function register. The same window register mechanism may be available in the UCIe link DVSEC function of the upstream port of the other endpoint to access the UCIe retimer registers on the far-end retimer.

[0101] FIG. 19A is a representation of an example window register 1900a for use by software. The window registers may be located on upstream or downstream components connected to the adapter of the UCIe die connected to the retimer. These window registers may be part of the standard link DVSEC functionality implemented by the upstream or downstream components. Such registers may include, by way of example, an SB mailbox address register 1905 that defines the target register, configuration or memory, the register locator number of that memory, the address offset, the opcode to be used on the sideband bus, etc. The window registers may include an SB mailbox data register (e.g., 1910) that provides data to be written or read from the device, a mailbox control register (e.g., 1915) that has register bits for triggering sideband accesses, and a mailbox semaphore register (e.g., 1920) that provides a simple mechanism for host software or a sideband debug agent to access the mailbox register set without conflict. In some implementations, the mailbox semaphore registers may utilize a simple req / gnt protocol where only two agents can compete for the mailbox register set.

[0102] Referring to FIG. 19B, a representation of a mailbox semaphore is shown. In one exemplary implementation, each of two agents may request ownership by writing to the Req0 / 1 bits. The hardware may assign ownership to one of these by setting the Gnt0 / 1 bits. The requesting agent polls the Gnt registers and knows it has won ownership. Once it wins the arbitration, it can safely use the mailbox register set, and once done, clears the Gnt bit by writing a 1 to it. The hardware may then grant ownership to any other agent that requests it. If a new agent requests ownership when someone else owns the mailbox register, it waits until the associated Gnt bit is cleared, and then it grants ownership to the pending agent. In some implementations, the opcode used to access the link DVSEC is a "Cfg read / write" that is 32 or 64 bits wide. The offsets accessed in sideband accesses for cfg read / write are shown in Figure 19B. To access the registers pointed to by the register locators, a sideband mechanism is used and the opcode used for this is "mem read / write" which is 32 or 64 bits wide. The DstID of the address used for the sideband access must point to one of the three (maximum) register locator regions. 000 indicates the register locator 0 related register block, 001 indicates the register locator 1 related register block, etc.

[0103] 20 depicts an example register locator register 1820. In one example, bits 5:3-2005 of register 1820 indicate which register locator provides access to which set of additional register blocks. For example, if register locator 0 has bits 5:3 as 0h, among other examples, then the UCIe D2D / PHY related registers can be accessed by accessing the register locator 0 field.

[0104] As mentioned above, retimers may be utilized to implement rack / pod level partitioning using the CXL 2.0 (or later) protocol. In one specific illustrated example, at the rack level, multiple compute nodes (e.g., in a virtual hierarchy) from different compute chassis connect to a CXL switch that connects to multiple CXL accelerators and / or type 3 memory devices that may be placed in one or more separate drawers. In this example, each host may be implemented in a compute drawer, and each compute drawer connects to the switch using an optical connection running the CXL protocol through a UCIe-based optical tile. The switch may have co-packaged optics where the optical tile connects to the main switch die using UCIe, and on the other side, a PCIe / CXL physical interconnect is used to connect to the accelerator / memory devices, among many other exemplary implementations where the UCIe link allows off-package connectivity.

[0105] It should be noted that the above apparatuses, methods, and systems may be implemented in any of the electronic devices or systems described above. As a specific example, the following figures provide an exemplary system for utilizing the solutions described herein. As the following systems are described in more detail, a number of different interconnects are disclosed, described, and reviewed from the above description. Although some of the above examples have been based on CXL or PCIe-based interconnects and devices, it should be understood that the above-described solutions and functions may be equally readily applied to other interconnects or protocols used to couple sockets, packages, boards, etc. within various computing platforms. As will be readily apparent, the above-described advances and UCIe-based protocols may be applied to any of the interconnects, fabrics, or architectures described herein, along with other comparable interconnects, fabrics, or architectures not expressly named or illustrated herein.

[0106] Referring to Figure 21, an embodiment of a block diagram for a computing system including a multi-core processor is shown. Processor 2100 includes any processor or processing device, such as a microprocessor, embedded processor, digital signal processor (DSP), network processor, handheld processor, application processor, co-processor, system on chip (SOC), or other device for executing code. In one embodiment, processor 2100 includes at least two cores, namely cores 2101 and 2102, which may include asymmetric cores or symmetric cores (illustrated embodiment). However, processor 2100 may include any number of processing elements, which may be symmetric or asymmetric.

[0107] In one embodiment, a processing element refers to hardware or logic that supports a software thread. Examples of hardware processing elements include a thread unit, a thread slot, a thread, a processing unit, a content, a context, a context unit, a logical processor, a hardware thread, a core, and / or any other element capable of maintaining a processor state, such as an execution state or an architectural state. In other words, in one embodiment, a processing element refers to any hardware capable of being associated with code, such as a software thread, an operating system, an application, or independently other code. A physical processor (or processor socket) typically refers to an integrated circuit that potentially contains any number of other processing elements, such as cores or hardware threads.

[0108] A core often refers to logic located on an integrated circuit capable of maintaining independent architectural states, where each independently maintained architectural state is associated with at least some dedicated execution resources. In contrast to a core, a hardware thread generally refers to any logic located on an integrated circuit capable of maintaining independent architectural states, where the independently maintained architectural states share access to the execution resources. As will be appreciated, the nomenclature boundaries between hardware threads and cores overlap when certain resources are shared and other resources are dedicated to architectural states. However, often cores and hardware threads are viewed by the operating system as individual logical processors, and the operating system can schedule operations on each logical processor separately.

[0109] As shown in FIG. 21, a physical processor 2100 includes two cores, namely, cores 2101 and 2102. Here, cores 2101 and 2102 are considered to be symmetric cores, i.e., cores with the same configuration, functional units, and / or logic. In another embodiment, core 2101 includes an out-of-order processor core and core 2102 includes an in-order processor core. However, cores 2101 and 2102 may be individually selected from any type of core, such as native cores, software-managed cores, cores adapted to execute a native instruction set architecture (ISA), cores adapted to execute a translated instruction set architecture (ISA), co-designed cores, or other known cores. In a heterogeneous core environment (i.e., asymmetric cores), some form of translation, such as binary translation, may be utilized to schedule or execute code on one or both cores. However, for further explanation, the functional units shown in core 2101 are described in more detail below. In the illustrated embodiment, the units in core 2102 operate in a similar manner.

[0110] As shown, core 2101 includes two hardware threads 2101a and 2101b, which may also be referred to as hardware thread slots 2101a and 2101b. Thus, in one embodiment, in some cases, a software entity, such as an operating system, views processor 2100 as four separate processors, i.e., four logical processors or processing elements capable of simultaneously executing four software threads. As alluded to above, a first thread may be associated with architecture state register 2101a, a second thread may be associated with architecture state register 2101b, a third thread may be associated with architecture state register 2102a, and a fourth thread may be associated with architecture state register 2102b. Here, each of the architecture state registers (2101a, 2101b, 2102a, and 2102b) may be referred to as a processing element, thread slot, or thread unit, as described above. As shown, architecture state registers 2101a are replicated in architecture state registers 2101b so that separate architecture state / context can be stored for logical processor 2101a and logical processor 2101b. In core 2101, other smaller resources such as instruction pointers and renaming logic in allocator and renamer block 2130 may also be replicated for threads 2101a and 2101b. Some resources such as reorder buffers, ILTB 2120, load / store buffers, and queues in reorder / retirement unit 2135 may be shared through partitioning. In some cases, other resources such as general-purpose internal registers, page table base registers, low-level data cache and data-TLB 2115, execution unit 2140, and portions of out-of-order unit 2135 are fully shared.

[0111] Processor 2100 often includes other resources. These may be fully shared or shared through partitioning, or may be dedicated to processing elements. FIG. 21 shows an embodiment of just an example processor along with example logical units / resources of the processor. Note that the processor may include or omit any of these functional units, as well as any other known functional units, logic, or firmware not shown. As shown, core 2101 includes a simplified typical out-of-order (OOO) processor core. However, different embodiments may utilize an in-order processor. The OOO core includes a branch target buffer 2120 for predicting branches to be executed / taken, and an instruction translation buffer (I-TLB) 2120 for storing address translation entries for instructions.

[0112] The core 2101 further includes a decode module 2125 coupled to the fetch unit 2120 for decoding the fetched elements. In one embodiment, the fetch logic includes separate sequencers associated with each of the thread slots 2101a, 2101b. Typically, the core 2101 is associated with a first ISA that defines / specifies instructions executable on the processor 2100. Often, the machine code instructions that are part of the first ISA include a portion of the instruction (called an opcode) that references / specifies the instruction or operation to be performed. The decode logic 2125 includes circuitry that recognizes these instructions from their opcodes and passes the decoded instructions to the pipeline for processing defined by the first ISA. For example, as described in more detail below, in one embodiment, the decoder 2125 includes logic designed or adapted to recognize unique instructions, such as transactional instructions. As a result of recognition by the decoder 2125, the architecture or the core 2101 takes a predefined unique action to perform the task associated with the appropriate instruction. It is important to note that any of the tasks, blocks, operations, and methods described herein may be executed in response to a single or multiple instructions, some of which may be new or old. Note that in one embodiment, the decoder 2126 recognizes the same ISA (or a subset thereof). Alternatively, in a heterogeneous core environment, the decoder 2126 recognizes a second ISA (a subset of the first ISA, or a separate ISA).

[0113] In one example, the allocator and renamer block 2130 includes an allocator for reserving resources such as a register file for storing instruction processing results. However, in some cases, threads 2101a and 2101b are capable of out-of-order execution, and the allocator and renamer block 2130 also reserves other resources such as a reorder buffer for tracking instruction results. The unit 2130 may include a register renamer for renaming program reference registers / instruction reference registers to other registers within the processor 2100. The reorder / retirement unit 2135 includes components such as the reorder buffer, load buffer, etc. described above, and stores buffers to support out-of-order execution and then support in-order retirement of instructions executed out-of-order.

[0114] In one embodiment, the scheduler and execution units block 2140 includes a scheduler unit for scheduling instructions / operations on the execution units. For example, floating point instructions are scheduled on ports of execution units that have an available floating point execution unit. Register files associated with these execution units are also included for storing information instruction processing results. Exemplary execution units include floating point execution units, integer execution units, jump execution units, load execution units, store execution units, and other known execution units.

[0115] A low level data cache and data translation buffer (D-TLB) 2150 is coupled to the execution unit 2140. The data cache stores recently used / operated on multiple elements, such as multiple data operands potentially held in multiple memory coherency states. The D-TLB is for storing recent virtual / linear to physical address translations. As a specific example, the processor may include a page table structure for dividing the physical memory into multiple virtual pages.

[0116] Here, the cores 2101 and 2102 share access to a higher level or further out cache, such as a second level cache associated with the on-chip interface 2110. Note that "higher level" or "further out" refers to cache levels going up from the execution units or further away from the execution units. In one embodiment, the higher level cache is a last level data cache, i.e., the last cache in the memory hierarchy on the processor 2100, such as a second or third level data cache. However, the higher level cache is not so limited, as it may be associated with or include an instruction cache. Rather, a trace cache, which is a type of instruction cache, may be coupled after the decoder 2125 to store recently decoded traces. Here, in some cases, an instruction refers to a macroinstruction (i.e., an overall instruction recognized by a decoder), which can be decoded into several microinstructions (micro-operations).

[0117] In the illustrated configuration, the processor 2100 also includes an on-chip interface module 2110. Historically, a memory controller, described in more detail below, has been included in a computing system external to the processor 2100. In this scenario, the on-chip interface 2110 is for communicating with devices external to the processor 2100, such as a system memory 2175, a chipset (which often includes a memory controller hub for connecting to the memory 2175 and an I / O controller hub for connecting to peripheral devices), a memory controller hub, a northbridge, or other integrated circuits. In this scenario, the bus 2105 may include any known interconnect, such as a multi-drop bus, a point-to-point interconnect, a serial interconnect, a parallel bus, a coherent (e.g., cache coherent) bus, a layered protocol architecture, a differential bus, and a GTL bus. The bus 2105, interconnect, and fabric interconnecting elements in the system of FIG. 21, as well as other links, may be implemented using UCIe according to the principles described above. In some implementations, one or more of the elements shown in the system of Figure 21 may be provided on the same package and interconnected. In some examples, some of the interconnected elements may be on different packages and may be coupled by off-package links, among other examples.

[0118] Memory 2175 may be dedicated to processor 2100 or shared with other devices in the system. Common examples of types of memory 2175 include DRAM, SRAM, non-volatile memory (NV memory), and other known storage devices. It should be noted that devices 2180 may include a graphics accelerator, graphics processor, or graphics card coupled to a memory controller hub, data storage coupled to an I / O controller hub, a wireless transceiver, a flash device, an audio controller, a network controller, or other known devices.

[0119] However, nowadays, more logic and devices are integrated on a single die such as a SOC, so each of these devices can be integrated on the processor 2100. For example, in one embodiment, a memory controller hub is provided in the same package and / or die as the processor 2100. Here, a portion of the core 2110 (the on-core portion) includes one or more controllers for interfacing with other devices such as memory 2175 or a graphics device 2180. A configuration including interconnects and controllers for interfacing with such devices is often referred to as an on-core (or uncore configuration). As an example, the on-chip interface 2110 includes a ring-type interconnect for on-chip communication and a high-speed serial point-to-point link 2105 for off-chip communication. However, in a SOC environment, much more devices such as network interfaces, co-processors, memory 2175, graphics processor 2180, and any other known computing device / computer interface can be integrated on a single die or integrated circuit to provide a small form factor with high functionality and low power consumption.

[0120] In one embodiment, the processor 2100 may execute compiler code 2177, optimization code 2177, and / or translator code 2177 to compile, translate, and / or optimize application code 2176 to support or interface with the apparatus and methods described herein. A compiler often includes a program or set of programs for translating source text / code into target text / code. Typically, compilation of program / application code using a compiler occurs in multiple phases and passes to convert high-level programming language code into low-level machine or assembly language code. However, for simple compilation, a single pass compiler may be utilized. A compiler may utilize any known compilation technique and perform any known compiler operations such as lexical analysis, preprocessing, parsing, semantic analysis, code generation, code transformation, and code optimization.

[0121] Larger compilers often include multiple phases, but in most cases these phases are included in two general phases: (1) the front-end, i.e., generally where syntactic processing, semantic processing, and some transformation / optimization may occur, and (2) the back-end, i.e., generally where parsing, transformation, optimization, and code generation occur. Some compilers refer to the middle, which describes a blurring of the description between the front-end and back-end of the compiler. As a result, references to insertion, association, generation, or other operations of the compiler may occur in any of the above phases or passes, as well as in any other known phases or passes of the compiler. By way of illustration, a compiler may optionally insert operations, calls, functions, etc. during one or more phases of compilation (e.g., inserting a call / operation in the front-end phase of compilation and then translating the call / operation into lower-level code during the translation phase). Note that compiler code or dynamic optimization code may insert such operations / calls during dynamic compilation and optimize the code for execution during run-time. As an illustrative example, binary code (already compiled code) may be dynamically optimized during run-time, where program code may include dynamically optimized code, binary code, or a combination thereof.

[0122] Similar to a compiler, a translator, such as a binary translator, statically or dynamically translates code to optimize and / or translate the code. Thus, a reference to running code, application code, program code, or other software environment may refer to (1) dynamically or statically running a compiler program, optimizing code optimizer, or translator to compile program code, maintain software structure, perform other operations, optimize code, or translate code, (2) running main program code, such as optimized / compiled application code, that includes operations / calls, (3) running other program code, such as libraries, associated with the main program code to maintain software structure, perform other software-related operations, or optimize code, or (4) any combination of (1)-(3).

[0123] Referring now to FIG. 22, a block diagram of a second system 2200 according to an embodiment of the present disclosure is shown. As shown in FIG. 22, the multiprocessor system 2200 is a point-to-point interconnect system and includes a first processor 2270 and a second processor 2280 coupled via a point-to-point interconnect 2250. The links, buses, interconnects, and fabrics that interconnect elements in the system may be implemented using UCIe according to the principles described above. In some implementations, one or more of the elements shown in the system of FIG. 22 may be provided on the same package and interconnected. In some examples, some of the interconnected elements may be on different packages and coupled by off-package links, among other examples. Each of the processors 2270 and 2280 may be some version of a processor. In one embodiment, 2252 and 2254 are part of a serial point-to-point coherent interconnect fabric, such as a high-performance architecture. As a result, the solutions described herein may be implemented in a UPI or other architecture.

[0124] Although shown with only two processors 2270, 2280, it should be understood that the scope of the disclosure is not so limited. In other embodiments, there may be one or more additional processors within a given processor.

[0125] Processors 2270 and 2280 are shown to include integrated memory controller units 2272 and 2282, respectively. Processor 2270 also includes point-to-point (PP) interfaces 2276 and 2278 as part of its bus controller unit, and similarly, second processor 2280 includes PP interfaces 2286 and 2288. Processors 2270, 2280 may exchange information using PP interface circuits 2278, 2288 via point-to-point (PP) interface 2250. As shown in FIG. 22, IMCs 2272 and 2282 couple the processors to memory, i.e., memory 2232 and memory 2234, respectively, which may be portions of main memory locally attached to the respective processors.

[0126] Processors 2270, 2280 exchange information with chipset 2290 via separate PP interfaces 2252, 2254 using point-to-point interface circuits 2276, 2294, 2286, 2298, respectively. Chipset 2290 also exchanges information with high performance graphics circuit 2238 via interface circuit 2292 through high performance graphics interconnect 2239.

[0127] A shared cache (not shown) may be included within either processor or external to both processors, but connected to the processors via the PP interconnect, allowing local cache information of either processor or both processors to be stored in shared memory when the processors are placed into a low power mode.

[0128] Chipset 2290 may be coupled to a first bus 2216 via an interface 2296. In one embodiment, first bus 2216 may be a bus such as a Peripheral Component Interconnect (PCI) bus, or a PCI Express bus or another third generation I / O interconnect bus, although the scope of the disclosure is not so limited.

[0129] As shown in FIG. 22, various I / O devices 2214 are coupled to a first bus 2216 along with a bus bridge 2218, which couples the first bus 2216 to a second bus 2220. In one embodiment, the second bus 2220 includes a low pin count (LPC) bus. In one embodiment, various devices are coupled to the second bus 2220, including, for example, a keyboard and / or mouse 2222, a communication device 2227, and a storage unit 2228, such as a disk drive or other mass storage device, which often includes instructions / code and data 2230. Additionally, an audio I / O 2224 is shown coupled to the second bus 2220. It should be noted that other architectures are possible. The component architectures and interconnect architectures involved are various. For example, instead of the point-to-point architecture of FIG. 22, the system could also implement a multi-drop bus or other such architecture.

[0130] While the solutions have been described herein with respect to a limited number of embodiments, those skilled in the art will appreciate numerous modifications and variations therefrom, and it is intended that the appended claims cover all such variations and modifications that fall within the true spirit and scope of the present disclosure.

[0131] A design may go through various stages from creation to simulation to fabrication. Data representing the design may represent the design in a number of ways. First, hardware description languages ​​or other functional description languages ​​may be used to represent the hardware, as is useful in simulation. In addition, circuit level models with logic and / or transistor gates may be generated at some stages of the design process. Furthermore, most designs reach a data level at some stage that represents the physical placement of various devices in the hardware model. If conventional semiconductor fabrication techniques are used, the data representing the hardware model may be data specifying the presence or absence of various features on different mask layers of the mask used to fabricate the integrated circuit. In any representation of the design, the data may be stored in the form of any machine-readable medium. Memory, or magnetic or optical storage such as a disk, may be machine-readable media that store such information transmitted via light or radio waves that are modulated or otherwise generated to transmit information. When an electrical carrier wave indicating or carrying the code or design is transmitted, a new copy is made to the extent that copying, buffering or transmission of the electrical signal is performed. Thus, a communications provider or network provider may store, at least temporarily, an article such as information encoded in a carrier wave on a tangible, machine-readable medium, embodying techniques according to embodiments of the present disclosure.

[0132] A module as used herein refers to any combination of hardware, software, and / or firmware. As an example, a module includes hardware such as a microcontroller associated with a non-transitory medium that stores code adapted to be executed by the microcontroller. Thus, in one embodiment, reference to a module refers to hardware specifically configured to recognize and / or execute code held on the non-transitory medium. Furthermore, in another embodiment, the use of a module refers to a non-transitory medium that includes code that is specifically adapted to be executed by a microcontroller to perform a number of predefined operations. As may be expected, in yet another embodiment, the term module (in this example) may refer to a combination of a microcontroller and a non-transitory medium. Often, the boundaries of multiple modules that are shown as separate generally vary and potentially overlap. For example, a first module and a second module may share hardware, software, firmware, or a combination thereof, while potentially maintaining some independent hardware, software, or firmware. In one embodiment, the use of the word "logic" includes hardware such as transistors, registers, or other hardware such as programmable logic devices.

[0133] In one embodiment, the use of the phrase "configured to" refers to arranging, putting together, manufacturing, offering to sell, importing, and / or designing an apparatus, hardware, logic, or element to perform a specified or determined task. In this example, an inoperative apparatus or element thereof is still "configured" to perform said specified task if it is designed, coupled, and / or interconnected to perform said specified task. As a purely illustrative example, a logic gate may provide a 0 or a 1 during operation. However, a logic gate "configured" to provide an enable signal to a clock does not include any underlying logic gate that may provide a 1 or a 0. Instead, the logic gate is coupled in some manner such that a 1 or 0 output enables the clock during operation. Again, note that the use of the term "configured" does not require an operation, but instead focuses on the hidden state of the apparatus, hardware, and / or element. In a hidden state, the device, hardware and / or element is designed to perform a specific task when the device, hardware and / or element is operational.

[0134] Additionally, in one embodiment, the use of the phrases "to," "capable of / to," and / or "operable to" refers to some device, logic, hardware, and / or element that is designed to enable use of the device, logic, hardware, and / or element in a specified manner. It should also be noted above that in one embodiment, the use of the words "to," "capable of," or "operable" refers to the hidden state of the device, logic, hardware, and / or element, where the device, logic, hardware, and / or element is not operational but is designed to enable use of the device in a specified manner.

[0135] Values, as used herein, include any known representation of a number, state, logic state, or binary logic state. Often the use of logic levels, logic values, or logic values, also referred to as "ones" and "zeros," simply represent binary logic states. For example, a one refers to a high logic level and a zero refers to a low logic level. In one embodiment, a storage cell, such as a transistor cell or a flash cell, may be capable of holding a single logic value or multiple logic values. However, other representations of values ​​in computer systems are used. For example, the number ten in the decimal system may also be represented by the binary value 1010 and the letter A in the hexadecimal system. Thus, values ​​include any representation of information that can be held in a computer system.

[0136] Further, a state may be represented by a value or a portion of a value. As an example, a first value, such as a logic one, may represent a default or initial state, while a second value, such as a logic zero, may represent a non-default state. Additionally, in one embodiment, the terms reset and set refer to default and updated values ​​or states, respectively. For example, a default value potentially includes a high logic value, i.e., reset, while an updated value potentially includes a low logic value, i.e., set. It should be noted that any combination of multiple values ​​may be utilized to represent any number of states.

[0137] The above-described methods, hardware, software, firmware, or code embodiments may be implemented via instructions or codes stored on a machine-accessible, machine-readable, computer-accessible, or computer-readable medium executable by a processing element. A non-transitory machine-accessible / readable medium includes any mechanism that provides (i.e., stores and / or transmits) information in a form readable by a machine, such as a computer or electronic system. For example, non-transitory machine-accessible media include random access memory (RAM), such as static RAM (SRAM) or dynamic RAM (DRAM), ROM, magnetic or optical storage media, flash memory devices, electronic storage devices, optical storage devices, acoustic storage devices, other types of storage devices for holding information received from a transitory (propagated) signal (e.g., carrier wave, infrared signal, digital signal), and the like, and these non-transitory machine-accessible media should be distinguished from non-transitory media that can receive information from these media.

[0138] The instructions used to program the logic to carry out the embodiments of the present invention may be stored within a memory in the system, such as a DRAM, cache, flash memory, or other storage. Additionally, the instructions may be distributed over a network or using other computer-readable media. Thus, a machine-readable medium may include any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer), including but not limited to tangible machine-readable storage such as floppy disks, optical disks, compact disks, read-only memories (CD-ROMs), and magnetic-optical disk read-only memories (ROMs), random access memories (RAMs), erasable programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs), magnetic or optical cards, flash memories, or the like used in transmitting information over the Internet via electrical, optical, acoustic, or other forms of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.). Thus, a computer-readable medium includes any type of tangible machine-readable medium suitable for storing or transmitting electronic instructions or information in a form readable by a machine (e.g., a computer).

[0139] The following examples relate to embodiments according to the present disclosure.

[0140] Throughout this specification, a reference to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with that embodiment is included in at least one embodiment of the disclosure. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" in various places throughout this specification do not necessarily all refer to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0141] In the foregoing specification, the detailed description has been given with reference to certain exemplary embodiments. It will be apparent, however, that various modifications or changes may be made thereto without departing from the broader spirit and scope of the invention as set forth in the appended claims. Accordingly, the specification and drawings should be regarded in an illustrative rather than a restrictive sense. Moreover, the above use of embodiment and other exemplary language does not necessarily refer to the same embodiment or the same example, but may refer to different individual embodiments and potentially the same embodiment.

Claims

1. A retimer, a first port coupled to a die via a first interconnect, the first interconnect including a defined set of lanes and utilizing a first communications technology, the die being disposed on a first package including the retimer; a second port coupled to another retimer via a second interconnect, the second interconnect utilizing a different second communication technology, the second retimer being located on a different second package; 23. An apparatus comprising a retimer including:

2. The device of claim 1 , wherein the second communication technology allows for a longer physical reach than the first communication technology.

3. 3. The apparatus of claim 1, wherein the first port includes a die-to-die adapter and a physical layer circuit, the die-to-die adapter ensuring successful data transfer across the first interconnect.

4. The physical layer circuit includes: a first number of sideband conductors carrying data for use in training and managing the link; a second number of main band conductors implementing a main data path of the link, the second number of main band conductors including a forward clock conductor, a enable lane conductor, and a number of data lane conductors; The apparatus of claim 3 , comprising:

5. 2. The apparatus of claim 1 , wherein the second retimer connects to a second die on the second package via a third interconnect, and the first die and the second die are communicatively coupled via a communication channel that includes the first interconnect, the second interconnect, and the third interconnect.

6. The apparatus of claim 5 , wherein the retimer communicates with the second retimer to enforce the same configuration of communication channels on the first package and the second package.

7. 7. The apparatus of claim 6, wherein the same configuration includes one or more of the same link width, the same speed, or the same Flit format used in both the first package and the second package of the communication channel.

8. 2. The apparatus of claim 1, wherein a Flit in a format defined according to a specific underlying protocol is received at the first port from the first die, and the retimer transfers data of the Flit to the second retimer via the second interconnect.

9. 9. The apparatus of claim 8, wherein the format defines one or more of error correction information or error correction information to be included in the Flit, and the retimer includes other error detection information or other error correction information in the data forwarded to the second retimer based on the second communication technology.

10. 9. The apparatus of claim 8, wherein the format is selected from a plurality of different Flit formats supported by the first die, the plurality of different Flit formats including a Peripheral Component Interconnect Express (PCIe) Flit format, a Compute Express Link (CXL) Flit format, and a Low mode format.

11. 2. The apparatus of claim 1, wherein the retimer further comprises a receive buffer and a flow control circuit that implements a flow control scheme for Flits sent from the first die to the retimer over the first interconnect.

12. 12. The apparatus of claim 11, wherein the flow control scheme includes providing credits to the first die, each of the credits corresponding to a single Flit, and a Flit is not sent from the first die to the retimer without a corresponding credit.

13. The apparatus of claim 1 , wherein the second communication technology includes one of optical or millimeter wave.

14. receiving a Flit at a first retimer on a first package, the Flit being received from a die on the first package over a first link, the first link being implemented using a first technology and including a set of sideband lanes and a separate set of main band lanes; preparing data of the Flit to be transmitted to a second retimer on a second package over a second link, the second link being implemented using a different second technology; transmitting the data of the Flit from the first retimer to the second retimer over the second link; A method for providing the above.

15. The method of claim 14 , wherein the data is prepared by the retimer to maintain a reliability level for forwarding a Flit using the second technique.

16. 16. The method of claim 15, wherein the reliability level is maintained on the first link using a first forward error correction scheme and the reliability level is maintained on the second link using a second, different forward error correction scheme.

17. initializing the second link; receiving link configuration information from the die over the set of sideband lanes; transferring the link configuration information over a third link for utilization by a second die coupled to the second retimer by the third link to facilitate negotiation by the second die and the second die of a common configuration for the first link and the third link; The method of claim 14 further comprising:

18. A system comprising means for carrying out the method according to any one of claims 14 to 17.

19. 20. The system of claim 18, wherein the means includes a machine-readable storage medium having instructions stored thereon, the instructions being executable to cause a machine to perform at least a portion of the method of any one of claims 14 to 17.

20. a first device package including a first die and a first retimer; The first retimer comprises: a first port coupled to the first die via a first die-to-die link, the first die-to-die link including a defined set of lanes and utilizing a first communication technology; a second port coupled to a second retimer via an off-package interconnect, the off-package interconnect utilizing a different second communication technology, the second retimer being located on a different second device package; Including, the system.

21. further comprising the second device package; The second device package includes: A second die; and a second retimer coupled to the second die by a second die-to-die link; Including, the first die and the second die are communicatively coupled via the first die-to-die link, the off-package interconnect, and the second die-to-die link; 21. The system of claim 20.

22. 22. The system of claim 21, wherein the first device package and the second device package are included in a server computing system.

23. 22. The system of claim 21, wherein the first die includes a first processing device and the second die includes a second processing device.

24. the first port includes a die-to-die adapter and a physical layer circuit, the die-to-die adapter ensuring successful data transfer across the first interconnect; 24. A system according to any one of claims 20 to 23.

25. The physical layer circuit includes: a first number of sideband conductors carrying data for use in training and managing the link; a second number of main band conductors implementing a main data path of the link, the second number of main band conductors including a forward clock conductor, a enable lane conductor, and a number of data lane conductors; 25. The system of claim 24, comprising:

Images