This technology translates memory access between UALink and CXL.mem across heterogeneous interconnects, enabling scaling up of the memory layer of GPUs and accelerators. Inventors: Ronen Aharon Hyatt, Opal Hyatt, Ethan Sharon Hyatt, Gil Teeberger

Abstract

This enables scaling up the memory layer of GPUs and accelerators. [Solution] A method for enabling memory access between different protocol domains through translation between UALink UPLI and CXL.mem, wherein a UPLI request containing a Read command and the first physical address is received, the UPLI request is translated into a CXL.mem M2S request consisting of a MemRd* and the second physical address, and the CXL.mem M2S request is sent to the second entity. The translation includes address, tag, and opcode conversion between the UPLI domain and the CXL.mem domain.

Description

TECHNICAL FIELD

[0001] Cross-Reference to Related Applications This application claims priority to the following matters: U.S. Provisional Patent Application No. 63 / 991,122, filed on February 25, 2026; U.S. Provisional Patent Application No. 63 / 931,124, filed on December 4, 2025; U.S. Provisional Patent Application No. 63 / 906,709, filed on October 28, 2025; U.S. Provisional Patent Application No. 63 / 895,053, filed on October 7, 2025; U.S. Provisional Patent Application No. 63 / 874,393, filed on September 2, 2025; U.S. Provisional Patent Application No. 63 / 856,653, filed on August 3, 2025; U.S. Provisional Patent Application No. 63 / 826,342, filed on June 18, 2025; U.S. Provisional Patent Application No. 63 / 811,859, filed on May 25, 2025; and U.S. Provisional Patent Application No. 63 / 784,089, filed on April 5, 2025.

BACKGROUND ART

[0002] Ultra Accelerator Link (UALink) is an interconnect protocol that enables communication between accelerators and switches in high-performance computing environments. UALink defines a Protocol Level Interface (UPLI) that specifies channels, commands, and fields for memory read and write transactions between originators and completers.

[0003] Compute Express Link (CXL) is an interconnect protocol that enables cache-coherent memory access and high-bandwidth communication between hosts and devices. CXL defines sub-protocols such as CXL.mem, CXL.io, and CXL.cache, each corresponding to different communication patterns. CXL.mem defines Master-to-Slave (M2S) messages and Slave-to-Master (S2M) messages for memory access operations between hosts and memory devices.

SUMMARY OF THE INVENTION

[0004] Some of the following implementations describe translation systems and methods between UALink UPLI and CXL.mem to enable memory access across different protocol domains. Various implementations involve communicating with a first entity based on UPLI and a second entity based on CXL.mem, receiving a UPLI request from the first entity containing read commands and the first physical address, and translating the UPLI request into a subordinate (M2S) request from the CXL.mem master, consisting of MemRd* and the second physical address. The CXL.mem M2S request is then sent to the second entity. The translation includes address translation between different address spaces, tag translation between UPLI ReqTag values ​​and CXL.mem Tag values, opcode translation, and error propagation across protocol boundaries. This process may be performed by a resource provisioning unit (RPU), processor, switch, or other appropriate device.

[0005] In other implementations, the system consists of a processing unit with an UALink port, a memory controller connected to the first memory, and a CXL port connected to the second memory. The processing unit provides the first memory path from the UALink port through the memory controller to the first memory, and the second memory path from the UALink port through the CXL port to the second memory. The processing unit further consists of an RPU configured to receive UPLI requests from entities and forward the translated requests to the first memory via the first memory path or to the second memory via the second memory path. [Brief explanation of the drawing]

[0006] [Figure 1A] This shows an example of a system that functions as an UALink memory switch appliance or an UALink memory pool. [Figure 1B]This TFD example demonstrates a multi-entity memory access scenario where the GPU accesses memory via CHI conversion from UPLI to ARM. [Figure 2A] This shows an example of a system consisting of a processor that includes an UALink port. [Figure 2B] This shows an example of a system consisting of a processor with an UALink port and a DDR channel. [Figure 3A] This shows an example of a system consisting of a processor with UALink and ISoL ports. [Figure 3B] This shows an example of a TFD that translates a UPLI request into a request used by the processor's coherent interconnect. [Figure 4A] This example shows a system consisting of an RPU that converts UALink-based traffic and CXL.mem-based traffic. [Figure 4B] This shows an example of a TFD illustrating the translation between a UPLI request and a CXL.mem M2S Req MemRd. [Figure 5A] This example shows a system with an RPU that allows UALink-based entities to access CXL-based resources linked to the RPU. [Figure 5B] This shows an example of a TFD illustrating intent-based translation between UPLI and CXL.mem. [Figure 6A] This example shows a system where the processor is configured with UALink ports, allowing external entities to access memory resources mapped to the address space used by the processor's coherent interconnect. [Figure 6B] This example shows how two UPLI requests, which are transferred to different memory locations, are mapped to the address space used by the processor's coherent interconnect. [Figure 7A] This example shows a system consisting of a processor with a coherent interconnect, UALink port, and CXL RP. [Figure 7B] This example demonstrates how TFD converts two UPLI requests into a coherent interconnect request and a CXL.mem request. [Figure 8A] This example shows a system that converts between UALink-based traffic and CXL.mem traffic. [Figure 8B] This provides an example TFD demonstrating the conversion between UPLI and CXL.mem requests, optionally including speculative memory reads. [Figure 9A-B] This shows an example of translation between CXL.mem and UALink-based protocols. [Figure 10A] This shows an example of an RPU with a translatable cache between CXL.mem and UPLI. [Figure 10B] This demonstrates an example of an RPU without a cache, and is translated between CXL.mem and UPLI. [Figure 11] This demonstrates an example where the host CPU accesses various devices via CXL.mem to UALink. [Figure 12A] This example shows a system consisting of cables configured to run between CXL and NVLink. [Figure 12B] This shows an example TFD illustrating the translation between a CXL.mem M2S MemRd request and an NVLink read request. [Figure 13] This example demonstrates a system where inference context data is migrated between accelerator local memory and CXL memory devices across UALink and CXL protocol domains. [Figure 14A] This example shows a system that converts between UALink-based traffic and CXL.cache traffic when communicating with a host. [Figure 14B] This shows an example TFD illustrating the conversion between a UPLI request and a CXL.cache D2H RdCurr request. [Figure 14C]This shows an example TFD illustrating conversion between a UPLI request and a CXL.cache D2H RdShared request. [Figure 15A] This shows an example of a system that converts between UALink-based traffic and CXL.cache traffic when communicating with a device. [Figure 15B] This shows an example TFD illustrating conversion between a UPLI request and a CXL.cache H2D SnpCur request. [Figure 15C] This shows an example TFD illustrating conversion between a UPLI request and a CXL.cache H2D SnpData request. [Figure 16A] This shows an example of a system consisting of a processor and an RPU chiplet, where the chiplet performs conversion between UALink and CXL. [Figure 16B] This shows an example TFD that converts a UPLI request to a protocol used in the coherent interconnect of a processor. [Figure 17A] This shows an example of a system constituted by cables that perform conversion between UALink-based traffic and CXL-based traffic. [Figure 17B] This shows an example of an active optical cable consisting of an RPU that converts UALink-based traffic and CXL-based traffic. [Figure 18A] This shows an example of a system that converts UALink-based traffic and CXL-based traffic. [Figure 18B] This shows an example TFD illustrating conversion between a UPLI transaction including UPLI read response / data and a CXL transaction. [Figure 18C] This shows an example TFD illustrating conversion between a UPLI write response and a CXL transaction. [Figure 19A-B] This shows an example of an RPU having a cache that distinguishes requests from the UALink network and requests from a local CU when performing conversion to CXL.cache. [Figure 20A] This example shows a system consisting of CXL entities coupled to a host via CXL.cache. The host is coupled to an UALink entity and includes an RPU that converts between CXL and UALink. [Figure 20B] This shows an example of a TFD (Time Function Diagram) that demonstrates the conversion of a D2H request for CXL.cache to a UPLI request targeting a CXL entity to an UALink entity. [Figure 21A] This example shows a system where an accelerator with an RPU and cache is connected to another accelerator via a UALink switch, and then connected to the host via CXL.cache. [Figure 21B] This example demonstrates the cache miss and cache hit behavior when TFD converts a UPLI read request into a CXL.cache request. [Figure 22A] This shows an example of a system that converts CXL-based traffic to UALink-based traffic. [Figure 22B] This shows an example TFD illustrating the translation between a CXL transaction and a UPLI transaction, including the UPLI read response / data. [Figure 22C] This shows an example of a TFD illustrating the conversion between CXL and UPLI transactions, including the UPLI write response. [Figure 23A] This shows an example of a system that converts between UALink-based and CXL-based traffic. [Figure 23B] This shows an example of a TFD illustrating the conversion between a UPLI request ReqCmd (Read) and a CXL.io UIOMRd. [Figure 23C] This shows an example of a TFD illustrating the translation between a UPLI request's ReqCmd (Read) and a CXL.io MRd. [Figure 24A] This example shows a system where UALink-based resource consumers can access CXL-based resources linked to entities. [Figure 24B] This shows an example TFD illustrating the translation between a UALink UPLI read request and a CXL.io UIOMRd. [Figure 25A] This shows an example of a system that performs conversions between UALink-based protocols and PCIe-based protocols. [Figure 25B] This shows an example of a TFD illustrating the conversion between a UPLI request ReqCmd (Read) and a PCIe MRd. [Figure 25C] This shows an example of a TFD illustrating the conversion between a UPLI request ReqCmd (Read) and a PCIe UIOMRd. [Figure 26A] This example shows a system consisting of an accelerator or RPU configured to convert between UALink-based and PCIe-based traffic. [Figure 26B] This shows an example of a TFD illustrating the conversion between UALink-based and PCIe-based traffic. [Figure 27A] This example shows a system consisting of an accelerator or processor that converts between UALink-based ports and PCIe-based ports. [Figure 27B] This shows an example of a TFD illustrating the conversion between UALink-based and PCIe-based traffic. [Figure 28A] This shows an example of a system configured to enable PCIe from UALink. [Figure 28B] This shows an example of a TFD demonstrating PCIe communication on UALink. [Figure 29A] This shows an example of a system consisting of a UALink port and a processor coherent interconnect. [Figure 29B] This example shows how two UPLI requests are forwarded to different memory locations and mapped to the address space used by the processor's coherent interconnect, as demonstrated by the TFD. [Figure 30A]This example shows a system consisting of entities that are converted between UALink-based traffic and ISoL traffic. [Figure 30B] This shows an example of a TFD illustrating the conversion between UALink-based traffic and Intel UPI traffic. [Figure 31A] This example shows a system consisting of a processor / RPU that converts between UALink-based traffic and ISoL traffic. [Figure 31B] This shows an example of a TFD illustrating the conversion between UALink-based traffic and ISoL traffic. [Figure 32A] This example shows a system consisting of an RPU that converts between UALink and a coherent interconnect interface. [Figure 32B] This shows an example of a TFD illustrating address translation between UALink UPLI and ARM CHI ReadOnce. [Figure 33A] This example shows a system that performs conversion between UALink-based traffic and ARM CHI traffic. [Figure 33B] This example demonstrates an RPU that converts between UALink-based traffic and CHI traffic using an intermediate protocol based on ARM AMBA ACE-Lite. [Figure 34A] This example shows a system consisting of cables that convert between UALink-based and NVLink-based traffic. [Figure 34B] This shows an example of a transaction flow diagram (TFD) for cable-based conversion between UALink-based and NVLink-based protocols. [Figure 35] This shows an example of a processor consisting of an RPU, NVLink interface, and UALink port. [Figure 36]This example shows a system with an asymmetric bandwidth configuration between the RPU (composed of NVLink, UALink, and CXL ports) and a coherent interconnect. [Figure 37A] This example shows a system optimized for accelerator applications using the UALink interface, where the RPU includes more home nodes than request nodes. [Figure 37B] This example shows a system optimized for memory pool and memory switch applications that utilize the UALink interface, where the RPU has more request nodes than home nodes. [Figure 38A] This shows an example of a system using the NVLink interface with an accelerator-optimized configuration. [Figure 38B] This shows an example of a system configured as an NVLink-based memory pool. [Figure 39] This shows an example of a system where an IC package combines an input / output (IO) die, a compute die, and an RPU die. [Figure 40A] This shows an example of a system where a memory switch is connected to an IC package, an I / O die, and an RPU die. [Figure 40B] This shows an example of a TFD illustrating the conversion between CXL.mem and CXL.cache related to memory switches. [Figure 41A] This shows an example of a system consisting of memory switches or memory pools. [Figure 41B] This shows an example of a TFD illustrating the conversion between CXL.mem and CXL.cache related to memory pools. [Figure 42A] This shows an example of a system consisting of xPU-based multiprotocol switches. [Figure 42B] This shows an example of a TFD illustrating the protocol and address translation between UALink and CXL.cache. [Figure 43A]This example shows a system configured to translate between the first UALink traffic and the second UALink traffic. [Figure 43B] This example demonstrates how TFD converts the processor to operate as a UALink-based switch. [Figure 44] This example demonstrates a scalable GPU interconnect using an xPU-based AI switch. [Figure 45A] This shows an example of an xPU-based switch and a memory-attached system. [Figure 45B] This example demonstrates two UALink transactions through an xPU-based switch using a TFD. [Figure 46] This shows an example of a CPU with an UALink port. [Figure 47] This shows an example of a memory switch or AI switch implementation using two IC packages. [Figure 48] This shows an example of a processor consisting of a CXL endpoint port and an RPU. [Figure 49] This shows an example of a CPU with an UALink port. [Figure 50A] This example demonstrates how to pass CXL traffic or CXL-related traffic using carrier protocol encapsulation on a PHY based on IEEE 802.3 PMA. [Figure 50B] This shows an example of a TFD illustrating the mapping between the CXL.mem transaction layer and the UPLI transaction layer. [Figure 51A] This example shows a system where the host and device are coupled via a bridge, which performs conversion between carrier protocols using a PHY based on CXL and IEEE 802.3 PMA. [Figure 51B] This shows a TFD example demonstrating bidirectional CXL communication between a host and a device using UALink as an intermediate protocol. [Figure 52A]This shows an example of a system configured to enable CXL on UALink, for example, through CXL tunneling on UALink. [Figure 52B] This shows an example of a TFD that demonstrates CXL communication over UALink (e.g., CXL over UALink or CXL tunneling). [Figure 53A] This example shows how to encapsulate a 68-byte CXL frit into a UPLI transaction consisting of Write commands. [Figure 53B] This shows an example of a TFD that illustrates the translation used to encapsulate a 68-byte CXL frit into a UPLI Write command. [Figure 54A] This example shows how to encapsulate a 256-byte CXL frit into a UPLI WriteFull command. [Figure 54B] This shows an example of a TFD that illustrates a translation of encapsulating or packing a CXL.mem request message into a 256-byte CXL frit, and then further encapsulating it into a UPLI WriteFull command. [Figure 55A] This shows an example of a system that converts CXL frit to UPLI transaction layer messages. [Figure 55B] This shows an example of a TFD illustrating the tunneling of CXL frits on UPLI. [Figure 56A] This example demonstrates using the UPLI write message command to tunnel CXL frit. [Figure 56B] This example demonstrates TFD tunneling a CXL transaction using TCPI write message commands. [Figure 57A] This example shows a system where the host consists of a device with a first CXL port, a retimer, a bridge, and a second CXL port. [Figure 57B] This shows an example of a TFD demonstrating host-device CXL communication based on a PHY according to IEEE 802.3 PMA. [Figure 58A] This example shows a system in which a CXL host and CXL devices are connected via two bridges using UALink PHY. [Figure 58B] This shows an example of a TFD illustrating the translation of CXL frit and UALink frit. [Figure 59A] This document provides system configuration examples that enable UALink over Ethernet-based networks (UALink over standard Ethernet, UALink over UET, UALink over ESUN, or UALink over SUE). [Figure 59B] This document presents examples of TFDs demonstrating UALink-based communications, such as UALink over Ethernet, UALink tunneling over Ethernet, UPLI over Ethernet, or UPLI tunneling over Ethernet. [Figure 60A] This shows an example of a system that functions as an NVLink memory switch appliance or an NVLink memory pool. [Figure 60B] This example of a TFD demonstrates a multi-entity memory access scenario in which the GPU accesses memory mapped to the physical address space through the translation from NVLink to ARM CHI. [Figure 61A] This shows an example of a GPU / CPU coupled to an xPU, which consists of dies connected via an inter-chip interface. [Figure 61B] This shows an example of a custom accelerator consisting of NVLink Fusion chiplets. [Figure 62A] This example shows a system that converts between NVLink-based traffic and CHI-based coherent interconnect traffic. [Figure 62B] This shows an example of a TFD that converts an NVLink read request to a CHI ReadOnce request. [Figure 63A]This example shows a system that performs conversion between NVLink-based traffic and ARM CHI traffic. [Figure 63B] This example demonstrates an RPU that converts between NVLink traffic and CHI traffic using an intermediate protocol based on ARM AMBA ACE-Lite. [Figure 64A] This shows an example of a system that converts between NVLink traffic and CHI-based traffic. [Figure 64B] This shows an example of RPU conversion between NVLink traffic and CHI traffic. [Figure 65A] This example shows a TFD (Transaction Format) that converts an NVLink read request to a PCIe UIO read request, and then to an ARM CHI ReadOnce request. [Figure 65B] This example shows a TFD (Time Flow Digit) that converts an NVLink read request to a CXL.cache RdCurr request, and then to an ARM CHI ReadOnce request. [Figure 66A] This example shows a system consisting of an external entity, an optional NVLink switch, and a processor comprising an RPU with an NVLink interface, a Request Agent (RA) proxy, and a Home Agent (HA) proxy. [Figure 66B] This example shows a system where an NVLink chiplet (such as NVLink Fusion) is configured on a processor and translated between NVLink and CHI. [Figure 67A] This example shows a system consisting of an xPU, including an RPU that converts NVLink traffic and CHI traffic. [Figure 67B] This example shows a system consisting of entities combining NVLink and CXL ports with a CHI interface, which enables memory access through the processor's coherent interconnect. [Figure 68A]This example shows a system consisting of a processor with an NVLink chiplet connected to the processor's coherent interconnect via NVLink-C2C. [Figure 68B] This example demonstrates a system that combines xPUs and GPUs, using an RPU to translate between NVLink traffic and CHI-based traffic. [Figure 69A] This example shows a system that functions as a multiprotocol memory pool consisting of NVLink-based interfaces. [Figure 69B] This example of a TFD (Transaction Function Diagram) illustrates a memory access scenario involving multiple entities, where separate NVLink and UALink transactions utilize the same coherent interconnect infrastructure to access memory. [Figure 70] This example shows a heterogeneous computing system consisting of a combination of NVLink chiplets and accelerators based on the ARM mesh architecture. [Figure 71A] This example shows a system consisting of devices that convert between NVLink-based traffic and CXL.mem traffic. [Figure 71B] This shows an example of a TFD illustrating the conversion between NVLink-based traffic and CXL.mem traffic. [Figure 72A] This example shows a system consisting of RPUs with an NVLink interface, which allows the GPU to access resources linked to the RPU via CXL.mem. [Figure 72B] This shows an example of a TFD (Transaction Format) illustrating the conversion between NVLink-based protocols and CXL.mem, which is used for communication with CXL devices and CXL memory. [Figure 72C] This shows an example TFD illustrating the conversion between NVLink-based requests and CXL.mem requests. [Figure 73A] This shows an example of a system that performs conversion between NVLink-based protocols and the CXL protocol. [Figure 73B]This shows an example of a system that connects the RPU within an NVLink switch or NVSwitch with NVLink and CXL.mem entities. [Figure 73C] This shows an example of a system consisting of an xPU equipped with an NVLink Fusion chiplet. [Figure 74A] This example shows a system consisting of a processor with a coherent interconnect and an NVLink interface. [Figure 74B] This example shows a TFD where two NVLink read requests are forwarded to different memory locations via a coherent interconnect. [Figure 75A] This shows an example of a system consisting of a processor with an NVLink interface and a CXL RP. [Figure 75B] This example shows how TFD converts the first NVLink request to a coherent interconnect protocol and the second NVLink read request to a CXL.mem M2S M2S MemRd request. [Figure 76A] This example shows a system consisting of a processor with an NVLink interface, processing cores, LLC, CXL RP, and memory controller connected via a memory channel. [Figure 76B] This example shows a transaction flow diagram (TFD) illustrating the conversion of NVLink traffic to traffic compliant with the protocol used by the processor's coherent interconnect, using CXL.cache as the intermediate protocol. [Figure 77A] This example shows a system consisting of computers connected between a first interface that communicates according to an NVLink-based protocol and a second interface that communicates according to CXL.cache. [Figure 77B] This shows an example of a cable consisting of an NVLink port and a CXL port, with an RPU integrated into the cable. [Figure 77C]This shows an example TFD illustrating the conversion between NVLink-based requests and CXL.cache requests. [Figure 78A] This example shows a system consisting of CXL.cache and an xPU that converts RPUs between NVLink. [Figure 78B] This example demonstrates how TFD translates a D2H request from CXL.cache into an NVLink read request and then translates the NVLink response into an H2D data message from CXL.cache. [Figure 79A] This example shows a system consisting of xPU, RPU, cache, and local memory, and illustrates two access paths for a CXL.cache D2H request. [Figure 79B] TFD shows an example of two D2H request flows for CXL.cache. One is the initial flow served from local memory with cache state transitions, and the other is the second flow which is translated into an NVLink read request. [Figure 80A] This example shows a system consisting of computers that combine an NVLink-based interface with a CXL.cache interface. [Figure 80B] This shows an example of a TFD that demonstrates the conversion of NVLink read requests to H2D requests (including SnpCur) in CXL.cache. [Figure 80C] This example demonstrates how TFD converts NVLink read requests into H2D requests (including SnpData) in CXL.cache. [Figure 81A] This shows an example of a system that converts between NVLink-based and CXL-based traffic. [Figure 81B] This shows an example of a TFD illustrating the translation between NVLink requests and CXL.io requests. [Figure 82A] This shows an example of a system that converts between NVLink-based and CXL-based traffic. [Figure 82B]This shows an example TFD demonstrating the conversion between NVLink-based read requests and UIO TLP on CXL.io; [Figure 82C] This shows an example of a TFD illustrating the translation between an NVLink-based read request and the CXL.io MRd TLP. [Figure 83A] This example shows a system consisting of computers configured to translate between CXL.io traffic and NVLink-based traffic. [Figure 83B] This shows an example TFD illustrating the conversion between CXL.io read requests and NVLink-based read requests. [Figure 84A] This shows an example of a system that converts between NVLink-based and PCIe-based traffic. [Figure 84B] This shows an example of a TFD illustrating the conversion between NVLink read requests and PCIe UIOMRd. [Figure 84C] This shows an example of a TFD illustrating the conversion between NVLink read requests and PCIe MRd. [Figure 85A] This shows an example of a cable consisting of an RPU that converts between PCIe-based traffic and NVLink-based traffic. [Figure 85B] This shows an example of a TFD illustrating the conversion between PCIe-based requests and NVLink-based requests. [Figure 86A] This example shows a system where a CXL host and CXL devices are connected via two bridges and utilize NVLink PHY. [Figure 86B] This example of a TFD (Transaction Function Diagram) illustrates tunneling of CXL.io transactions between a host and a device via two bridges utilizing IEEE 802.3 and NVLink frit. [Figure 87A] This shows an example of a system configured to enable CXL over NVLink, such as CXL tunneling over NVLink. [Figure 87B] This shows an example of a TFD demonstrating CXL communication over NVLink (e.g., CXL over NVLink or CXL tunneling). [Figure 88A] This shows an example of a system configured to enable PCIe over NVLink, such as PCIe tunneling over NVLink. [Figure 88B] This shows an example of a TFD that demonstrates PCIe UIO communication (e.g., PCIe UIO over NVLink or PCIe UIO tunneling on NVLink). [Figure 89A] This shows an example of a processor consisting of an NVLink interface, processing cores, LLC, and coherent interconnect. [Figure 89B] This shows an example of a processor consisting of an NVLink interface and a DDR Channel. [Figure 90A] This shows an example of a processor consisting of an NVLink interface, processing cores, cache, and coherent interconnect. [Figure 90B] This example shows a TFD that translates NVLink read requests into read transactions for protocols used in the processor's coherent interconnection. [Figure 91A] This shows an example of a system consisting of a processor with an NVLink interface. [Figure 91B] This example of a TFD demonstrates two conversions: one from NVLink-based traffic to traffic compliant with the protocol used by the processor's coherent interconnect. [Figure 92A] This shows an example of a system that performs conversion between NVLink-based traffic and ISoL traffic. [Figure 92B] This shows an example of a TFD illustrating the conversion between NVLink-based traffic and Intel UPI traffic. [Figure 93A]This shows an example of a memory switch configured to provide memory to a combined entity. [Figure 93B] This shows an example of a TFD requesting NVLink access from an entity to memory. [Figure 94A] This shows an example of a system that implements an NVLink-based switch, i.e., an NVLink memory switch, using a CHI-based interconnect. [Figure 94B] This shows an example of a TFD (Total Function Display) demonstrating NVLink switching operation between entities via a switch interconnect. [Figure 95A] This shows an example of a system consisting of a processor with an NVLink-C2C interface. [Figure 95B] This shows an example of a system consisting of a CPU with an NVLink-C2C chiplet, such as NVLink Fusion. [Modes for carrying out the invention]

[0007] In various implementations, the system consists of the following components: A processor with a coherent interconnect connects processing cores to a memory controller, which in turn connects to memory channels capable of supporting at least 64GB of memory. The processor is configured to access memory using physical addresses in the physical address space (PAS) and to run an operating system (OS) that utilizes the virtual address space. A memory management unit (MMU) is configured to map addresses in the virtual address space to physical addresses in the PAS, allowing the OS to access memory. The resource provisioning unit (RPU) consists of Ultra Accelerator link-based ports (UALink-based ports) configured to communicate with entities connected to the device according to an UALink-based protocol. The RPU is further coupled to the coherent interconnect and configured to translate physical addresses related to the UALink-based protocol to physical addresses in the PAS. Physical address translation allows entities to access memory through the UALink-based ports, the coherent interconnect, and the memory controller. Address translation between physical addresses enables controlled access to system resources while allowing isolation between different address domains. Translation logic may support various mapping methods, including offset-based translation, page-table-based translation, and range-based translation. The RPU may include translation lookaside buffers (TLBs) and other caching mechanisms to optimize translation performance for frequently accessed address ranges.

[0008] In some implementations of the device, the UALink-based protocol conforms to the UALink protocol-level interface (UPLI), where the physical addresses associated with UPLI consist of network physical addresses (NPAs), and the physical addresses within the PAS consist of system physical addresses (SPAs) or host physical addresses (HPAs). A global or flat addressing model may be used, and the address space may include addresses used for memory access within a system domain, as well as UPLI-related addresses used for memory access in different system domains. Physical address translation is performed between physical addresses within that single address space. Alternatively, some implementations use physical address spaces such as the NPA space (when NPAs are used for memory access in different system domains) or the SPA space (when SPAs are used for memory access within a system domain), and physical address translation from NPA to SPA may also be performed. In some examples, an NPA may represent an address in a global or flat UALink network address space spanning multiple nodes or devices, or it may represent an address in the destination UALink accelerator referenced by the UPLI destination identifier. SPA or HPA (implementation choice) may represent the local addressing scheme used by the processor or system node (SN). The conversion from NPA to SPA / HPA may involve extracting routing information, processing node identifiers, or calculating address offsets to map fabric-side addresses to local memory locations.

[0009] In the device of claim 1, which is a partial implementation of the device, in addition to physical address translation, the RPU is further configured to translate between a first field conforming to the UALink-based protocol message format and a second field conforming to the message format of the protocol used in the coherent interconnect.

[0010] In some implementations of the device, the protocol used in the coherent interconnect is based on the Coherent Hub Interface (CHI-based protocol), and the RPU is further configured to translate read requests corresponding to UALink-based protocols into requests corresponding to CHI-based protocols with ReadOnce for non-cached data access and ReadShared for cacheable data access. The choice between ReadOnce and ReadShared may be determined by cache allocation hints, memory region attributes, or explicit indicators within the UPLI request. The RPU may further translate UPLI write requests into other CHI write opcodes based on write granularity and consistency requirements. This translation may also leverage CHI order rules and implement additional order execution logic if UALink order requirements exceed CHI provisions.

[0011] In some implementations of the device, the protocol used in the coherent interconnect is based on the Intel Coherent Processor Interconnect Protocol (ICPIP-based protocol) for scalable multiprocessors with a shared physical address space, and the RPU is further configured to translate read requests corresponding to UALink-based protocols into requests corresponding to ICPIP-based protocols, including the RdCur opcode. The RPU then maintains coherence state information for physical addresses in the PAS associated with the coherent interconnect. If the coherent interconnect is based on the UPI protocol, the coherence state information managed by the RPU may include cache state and similar cache coherence state. Examples of ICPIP include Intel's QPI, UPI, and future Intel coherent processor interconnect protocols. The translation of ICPIP to the RdCur opcode, such as the UPI RdCur opcode, may involve snoop response processing if the requested data exists in another processor cache. The RPU may include state tracking mechanisms to optimize subsequent access to the same cache line.

[0012] In some implementations of the device, the protocol used in the coherent interconnect is based on the Infinity Fabric protocol (IF-based protocol). The RPU is also configured to maintain the write order required by entities while translating write requests corresponding to UALink-based protocols into commands corresponding to IF-based protocols. Maintaining write order includes tracking write dependencies and enforcing the completion order specified in the UALink memory model. The RPU can translate UPLI write requests, including byte enablement indicating partial writes, into the appropriate Infinity Fabric write command type while maintaining producer-consumer ordering relationships. For example, the UPLI's 64-bit byte enablement field (OrigDataByteEn) allows whether to write individual bytes within a data beat during a write transfer, and the RPU can translate this into the appropriate Infinity Fabric write command type.

[0013] In some implementations of the device, the RPU is further configured to translate requests corresponding to UALink-based protocols into at least one message corresponding to the protocol used in the coherent interconnect, and at least one message to trigger a prefetch into the processor cache. The RPU can translate UPLI requests, such as UPLI prefetch hints carried in vendor-defined commands, into messages of the protocol used in the coherent interconnect, such as LLC prefetch RFO (LlcPrefRFO), LLC prefetch code (LlcPrefCode), LLC prefetch data (LlcPrefData), and opcodes for the Intradie Interconnect (IDI) based protocol. This is the protocol used in some Intel processor cores.

[0014] In some implementations of the device, the RPU is further configured to: translate tags associated with transactions corresponding to UALink-based protocols into tags used in the coherent interconnect, and maintain a mapping between transaction-associated tags and tags used in the coherent interconnect. The tag translation logic may accommodate different tag formats and sizes between the UPLI and coherent interconnect domains. Tags may be used to identify transactions, such as when supporting unprocessed requests through the RPU, or to convey characteristics related to messages and transactions, such as trace tags used for debugging and performance measurement, or authorization tags for security. Tags may be referred to by different names in different examples. For example, some ARM CHI implementations may call them Transaction Identifiers (TxnID). Mapping between UPLI tags and coherent interconnect tags may involve the use of on-silicon SRAM, Content Address Memory (CAM) or Ternary Content Address Memory (TCAM) structures, hash tables, or index arrays. The RPU can handle tag exhaustion scenarios by including flow control mechanisms to prevent new transactions when available tags are exhausted.

[0015] In some implementations, the RPU is further configured to: maintain a tag allocation table to track outstanding transactions from entities; allocate consistent interconnection tags from a pool of available tags when a request compliant with the UALink-based protocol is received; and release tags upon completion of the corresponding transaction. The tag allocation table can be resized to accommodate the maximum number of outstanding transactions allowed by the UALink specification and any configured limits. Tag pool management can implement various allocation techniques, such as round-robin, recently least used allocation, and priority-based allocation. The RPU may also monitor tag utilization to detect potential bottlenecks and include tag recycling logic to efficiently handle long-latency transactions.

[0016] In some implementations of the device, entities are configured to access memory using read / write requests compliant with UALink-based protocols, which are translated by the RPU. The processing core issues consistent interconnect requests to access resources associated with the entities, and the RPU is configured to further translate these into UALink-based protocol-compliant transactions, which target the entities. Bidirectional access capabilities can enable various computing paradigms, including memory pooling, memory sharing, resource partitioning, and heterogeneous computing. When the processing core accesses entity-attached resources such as high-bandwidth memory (HBM) or high-bandwidth flash (HBF) resources, the RPU may handle different memory attributes, cache policies, and sequencing requirements between the two domains. When translating consistent interconnect requests into UALink-based protocol-compliant transactions, protocol-specific adaptations may be included to maintain accuracy across domain boundaries.

[0017] In some implementations of the device, entities are configured in entity-attached memory. Furthermore, the RPU maps a portion of the entity-attached memory to the PAS, configuring it so that processing cores can access the entity-attached memory using load and store operations. Mapping entity-attached memory to the PAS may involve setting a memory window with specific attributes such as caching, write-bound behavior, and access permissions. The RPU may support dynamic remapping of the entity-attached memory region based on workload requirements and system configuration changes. Load and store operations from processing cores may adhere to memory ordering rules enforced by both the coherent interconnect and UPLI.

[0018] In some implementations, the RPU is further configured to enforce access control by comparing the physical address associated with the UALink-based protocol to a predetermined allowed address range for the entity and blocking transactions outside that range. The predetermined allowed address range can be configured by privileged software, firmware, or hardware configuration registers. The RPU may support different security contexts with different predetermined allowed address ranges for different entities or operating modes. Blocking unauthorized transactions may generate error responses, security exceptions, or log events for system monitoring and debugging purposes.

[0019] In some implementations of the device, the RPU is further configured to apply security filtering based on the inspection of transaction attributes related to UALink-based protocols, including requester identification and access rights, and selective permission or denial of transactions based on pre-configured security policies. Security filtering may also validate additional UPLI transaction attributes, such as vendor-defined commands and fields, virtual channel identifiers, traffic classes, and custom security tokens. Pre-configured security policies may be stored in secure storage within the RPU or loaded from trusted system firmware during initialization. The RPU may support dynamic policy updates under appropriate authentication and authorization mechanisms.

[0020] In some implementations of the device, the RPU is further configured as follows: it detects sequential access patterns of requests corresponding to UALink-based protocols received from entities and issues prefetch requests to retrieve pre-data via coherent interconnections and memory controllers. The prefetch mechanism may utilize various pattern detection algorithms, such as stride detection, stream detection, and machine learning-based prediction. The RPU maintains a prefetch buffer, stores prefetched data, and implements prefetch throttling to prevent memory bandwidth saturation. Prefetch requests may be marked with a lower priority than explicit memory access requests to minimize interference with explicit memory access.

[0021] In some implementations of the device, memory consists of dynamic random access memory (DRAM), and entities consist of graphics processing units (GPUs) or central processing units (CPUs), configured to access memory using UALink-based ports. The RPU also enables entities to access DRAM at cache line granularity. Entities such as GPUs and CPUs may utilize UALink ports for high-bandwidth memory access to memory resources connected to the processor. Optionally, cache line granular access may be aligned to standard cache line sizes such as 64 bytes, 128 bytes, or 256 bytes, enabling efficient data transfer between entities and DRAM. The RPU may support memory consistency maintenance, which involves reconciling the memory model of entities with that of the processor, with the RPU performing conversions between different consistency requirements. High-bandwidth memory access may be optimized by leveraging features such as memory interleaving, bank-aware scheduling, or quality control mechanisms that prioritize latency-sensitive or bandwidth-consuming access patterns from GPU or CPU entities.

[0022] In some implementations of the device, the RPU is further configured to reduce the number of requests for coherent interconnect transactions targeting contiguous or adjacent addresses, corresponding to UALink-based protocols. This reduces transaction overhead and improves memory bandwidth utilization. Request consolidation may consider factors such as address proximity, request type, and timing window when deciding which requests to combine. The RPU may include write-combined buffers for write requests and may also support read-coiling for sequential read patterns.

[0023] In some implementations of the device, the RPU is further configured to utilize an intermediate protocol selected from Peripheral Component Interconnect Express (PCIe) or Compute Express Link (CXL) when converting between UALink-based protocols and protocols used in the coherent interconnect.

[0024] In various implementations, the system consists of the following components: A processor with a coherent interconnect connects processing cores to a memory controller, which in turn connects to memory channels capable of supporting at least 64GB of memory. The processor is configured to access memory using physical addresses in the first physical address space (PAS1) and to run an operating system (OS) that utilizes the virtual address space. A memory management unit (MMU) is configured to map addresses in the virtual address space to physical addresses in PAS1, allowing the OS to access memory. The first and second resource provisioning units (RPUs) consist of first and second ultra-accelerator link-based ports (UALink-based ports), respectively, and are configured to communicate according to the UALink-based protocol. The first and second entities are connected to the device, and the first and second entities utilize their respective physical address spaces (PAS2, PAS3). The first and second RPUs are further connected to the coherent interconnect. Here, PAS1, PAS2, and PAS3 are distinct. The device then allows the first and second entities to access portions of the memory through the first and second UALink-based ports, coherent interconnects, and memory controllers.

[0025] In some implementations of the device, the first RPU is configured to translate physical addresses in PAS2 to physical addresses in PAS1, and the second RPU is configured to translate physical addresses in PAS3 to physical addresses in PAS1. This allows the first and second RPUs to access the memory of the first and second entities.

[0026] In some implementations of the device, the UALink-based protocol conforms to the UALink Protocol Level Interface (UPLI). In addition to physical address translation, the first and second RPUs are further configured to translate between the first field, which conforms to the UPLI message format, and the second field, which conforms to the second message format of the protocol used in the coherent interconnect.

[0027] In some implementations of the device, the protocol used in the coherent interconnect is based on the Coherent Hub Interface (CHI-based protocol), and at least the first and second RPUs are further configured to translate UPLI read requests into CHI-based requests with ReadOnce for non-cacheable data access and ReadShared for cacheable data access.

[0028] In some implementations of the device, the protocol used in the coherent interconnect is based on Intel's Coherent Processor Interconnect Protocol (ICPIP-based protocol) for scalable multiprocessors with a shared physical address space, with at least the first and second RPUs further configured to translate read requests corresponding to UPLI into requests corresponding to ICPIP-based carry instructions based on RdCur.

[0029] In some implementations of the device, the protocol used in the coherent interconnect is based on Infinity Fabric (IF-based). Furthermore, at least one of the first and second RPUs is further configured to translate UPLI-compatible write requests into IF-based write commands while maintaining the required write order for each entity.

[0030] In some implementations of the device, at least the first and second RPUs are further configured to convert tags associated with transactions corresponding to UALink-based protocols to tags used in the coherent interconnect, maintain the mapping between transaction-associated tags and tags used in the coherent interconnect, and convert response tags associated with the coherent interconnect to response tags related to UALink-based protocols.

[0031] In some implementations of the device, the first RPU maintains the first translation table for mapping addresses in PAS2 to addresses in PAS1, and the second RPU maintains the second translation table for mapping addresses in PAS3 to addresses in PAS1. Here, the first and second translation tables are different, isolating memory access from the first and second entities.

[0032] In some implementations of the device, the first RPU is configured to translate addresses in PAS2 to addresses in the first subset of PAS1, and the second RPU is configured to translate addresses in PAS3 to a second subset of PAS1, where the first and second subsets do not overlap.

[0033] In some implementations of the device, the first RPU is configured to translate at least some addresses in PAS2 to a shared subset in PAS1, and the second RPU is configured to translate at least some addresses in PAS3 to the same shared subset, allowing the first and second entities to access the shared memory region.

[0034] In some implementations of the device, PAS2 has a different size than PAS3, and PAS2 and PAS3 have different sizes than PAS1. Furthermore, the first and second RPUs are further configured to dynamically change address translation between PAS2 and PAS1, and between PAS3 and PAS1, based on memory allocation requests and reconfiguration commands.

[0035] Various implementations employ the following methods: running a processor consisting of a coherent interconnect that connects processing cores and a memory controller and communicates with a memory channel with a capacity of at least 64GB; the processor accessing memory using physical addresses in the physical address space (PAS) and running an operating system (OS) that utilizes a virtual address space; mapping addresses in the virtual address space to physical addresses in the PAS so that the OS can access memory; communicating with entities via UALink-based ports based on a protocol based on Ultra Accelerator Link (UALink-based protocol); and performing physical address translation from physical addresses related to the UALink-based protocol to physical addresses in the PAS; through physical address translation, entities can access memory through UALink-based ports, coherent interconnects, and memory controllers.

[0036] In some implementations of this method, the coherent interconnect utilizes a protocol based on the coherent hub interface (CHI-based protocol), which, in addition to performing physical address translation, (a) translates between a first field conforming to the UALink-based protocol message format and a second field conforming to the CHI-based protocol message format; and (b) translates UALink-based protocol read requests into CHI-based protocol requests that include ReadOnce for non-cached data access and ReadShared for cacheable data access.

[0037] Figure 1A shows an example of a system that functions as an UALink memory switch appliance or UALink memory pool, including an accelerator, GPU, xPU / MxPU, or memory switch ASIC, and is connected to two entities called Entity.1 / GPU.1 and Entity.2 / GPU.2. The accelerator / xPU includes a processing core and memory controller connected to a coherent interconnection, which may be CHI-based. The accelerator / xPU utilizes translations performed by the RPU between the UALink-based port and the coherent interconnect. The first RPU (RPU.1) may allow Entity.1 / GPU.1 to access the accelerator / xPU's memory resources, etc., mapped to the physical address space used by the coherent interconnect, through the first UALink port and coherent interconnection. Correspondingly, the second RPU (RPU.2) allows access to resources mapped to the physical address space used by the coherent interconnect, such as the accelerator / xPU's memory resources, through the second UALink port and coherent interconnection.

[0038] Figure 1B shows an example TFD illustrating a multi-entity memory access scenario, where the first and second entities / GPUs access memory mapped to one or more physical address spaces used by the coherent interconnect via UPLI-to-ARM CHI translation. Entity.1 / GPU.1 identifies the source entity, such as an accelerator or GPU, with ReqCmd(Read) and ReqSrcPhysAccID(a.1), identifies the destination entity / accelerator with ReqDstPhysAccID(b.1), and indicates the UPLI request address (Network Physical Address (NPA), etc., from the second physical address space) with ReqAddr(AS.2.1). RPU.1 receives the initial UPLI request from the first physical address space used by the coherent interconnect, including Opcode(ReadOnce) and Addr(AS.1.1) within the ARM address space. This is converted to a CHI request. Simultaneously or sequentially, Entity.2 / GPU.2 can initiate a second UPLI request (Req) from a third or second physical address space with ReqCmd(Read), ReqSrcPhysAccID(a.2), ReqDstPhysAccID(b.2), and ReqAddr(AS.3.1), representing the UPLI request address such as the network physical address (NPA). RPU.2 converts the second UPLI request, including Opcode(ReadOnce) and Addr(AS.1.2), to an ARM CHI REQ from the physical address space initially used by the coherent interconnect. Both transactions flow through the coherent interconnect to one or more home nodes, sending their respective ARM CHI REQ messages to one or more memory controllers, which assign Opcode(ReadNoSnp) and addresses Addr(AS.1.1) and Addr(AS.1.2), respectively.

[0039] The memory controller retrieves the requested data from memory and sends the first and second ARM CHI RDAT messages using the OPCODE (CompData). These represent data retrieved from addresses AS.1.1 and AS.1.2, respectively. RPU.1 translates the first ARM CHI RDAT message to Entity.1 / GPU.1 using RdRspSrcPhysAccID(b.1), RdRspDstPhysAccID(a.1), and RdRspData(*Data.1*) to create a UPLI read response / data (RdRsp). RPU.2 translates the second ARM CHI RDAT message to Entity.2 / GPU.2 using RdRspSrcPhysAccID(b.2), RdRspDstPhysAccID(a.2), and RdRspData(*Data.2*) to create a UPLI RdRsp. This example demonstrates how entities and GPUs can share access to the same memory through different RPUs that perform both UPLI and ARM CHI translation and physical address translation. Alternatively, this example can be viewed as two separate UPLI transactions accessing memory using the same coherent interconnect infrastructure, where entities such as accelerators and GPUs can access memory through a shared or separate address space and translate it to a shared coherent interconnect physical address space. Furthermore, response and read data paths may be implemented in other designs, for example, a memory controller sending data to a home node, which then sends it to its respective RPU, or a home node sending a response to an RPU, which then sends the data back to the RPU.

[0040] Figure 2A shows an example of a system consisting of a processor (such as an MxPU derived from an existing processor design) comprising processing cores, a last-level cache (LLC), and a coherent interconnect with a ring topology. Alternatively, the coherent interconnect may utilize other topology structures such as mesh, crossbar, or custom topology. The MxPU may include UALink ports, ISoL ports such as AMD Infinity Fabric, ARM CHI C2C, NVIDIA NVLink-C2C, Intel UPI, etc. The MxPU may also include a PCIe / CXL root port (PCIe / CXL RP) and can be coupled to memory such as DRAM via a memory controller or memory channel. According to UPLI, the UALink port can communicate with entities such as GPUs and CPUs, and optionally via a switch, the RPU performs physical address translation, allowing entities to access memory via the UALink port. The RPU shown in the figure is coupled to an on-chip ring interconnect via a coherent interconnect interface such as a ring-to-RPU (R2RPU). Alternatively, the RPU may be connected almost directly to the ring interconnect on the chip. Similarly, the illustrated ISoL ports are connected from the ring to the on-chip ring interconnect via a coherent interconnect interface such as ISoL (R2ISoL), and the PCIe / CXL root ports (RPs) are connected from the ring to the on-chip ring interconnect via a coherent interconnect interface such as PCIe / CXL (R2PCIe / CXL). The MxPU may be implemented as a monolithic die, as a chiplet within an IC package, for example using separate compute dies or I / O dies, or as a component on a board.

[0041] Figure 2B shows an example system in which the processor (e.g., MxPU) includes UALink ports, DDR channels, at least one optional PCIe / CXL RP, and at least one optional ISoL. UALink ports must not share the same physical address space and may be connected to entities, connected via switches as needed, and can communicate with each other according to UPLI.

[0042] Figure 3A shows an example of a system consisting of a processor using a ring topology structure (e.g., an MxPU derived from an existing processor design). Alternatively, the coherent interconnect may utilize other topology structures such as mesh, crossbar, or custom topology. The processor may also include ISoL ports such as ARM CHI C2C or Intel UPI, and a memory controller connected via memory channels such as DRAM. According to UPLI, the processor includes an UALink port that can communicate with entities such as GPUs and accelerators, and the RPU performs physical address translation, allowing the entities to access memory. The RPU shown in the figure is connected to the coherent interconnect via a ring-to-ISoL (R2ISoL) coherent interconnect interface. Alternatively, the RPU may be connected almost directly to the on-chip ring interconnect. Similarly, the illustrated ISoL port is connected to the on-chip ring interconnect via a Ring-to-ISoL (R2ISoL) coherent interconnect interface. Processors may be implemented as a monolithic die, as chiplets within an IC package utilizing separate compute and I / O dies, or as components on a circuit board.

[0043] Figure 3B shows an example of a TFD (Transaction Function Display) illustrating a UPLI request (Req) received from an entity such as a GPU or accelerator. Here, the RPU can translate the physical address (AS.2.1) contained in the UPLI request to the physical address (AS.1.1) used for memory access. The RPU can perform further translations, for example, translating from UPLI to the protocol used for the processor's coherent interconnect, and optionally sending the translated request to the home agent (also called the home node) or memory controller to request an address read (AS.1.1). In some examples, the requested data may be provided not from memory, but from a processor cache such as an LLC. The data is returned to the RPU through the processor's coherent interconnect, and the RPU provides a UPLI read response / data (RdRsp) to the requesting entity.

[0044] In environments where entities use different protocols but require coordinated access to shared resources, there are scenarios where the first entity communicating based on UALink UPLI (e.g., an accelerator) needs to access memory resources in conjunction with a second entity communicating according to CXL, such as CXL.mem. The RPU performs translation between UPLI and CXL.mem, facilitating memory operations, data transfer, and resource sharing across different protocol domains while maintaining the requirements of each protocol. The RPU can translate opcodes, commands, addresses, tags, and additional fields between UPLI and CXL.mem messages, and may even translate between different address spaces. For example, between the network physical address (NPA) space used by UALink-based traffic and the host physical address (HPA) space used by CXL-based traffic, or even between addresses within the same address space, such as the global address space, partitioned global address space (PGAS), pod address space, virtual pod address space, or fabric address space. The RPU may be implemented as a discrete component, as an IP block embedded in a processor, or as a chiplet within an IC package.

[0045] In various implementations, there are methods for translating Ultra Accelerator Link (UALink) protocol-level interface (UPLI) requests to Compute Express Link (CXL) requests, which include: communication with a first entity based on UPLI; communication with another entity according to CXL.mem; receiving a UPLI request from the first entity consisting of a read command and the first physical address; translating the UPLI request into a master-to-sublesson request in CXL.mem, including MemRd* and a second physical address (CXL.mem M2S Req MemRd*); and sending CXL.mem M2S Req MemRd* to the second entity. This translation allows an entity communicating according to UPLI to access memory resources in conjunction with an entity communicating according to CXL.mem. This method can be implemented in hardware, firmware, software, or a combination thereof, and may be performed by various devices such as processors, switches, bridges, RPUs, and semiconductor devices. MemRd* may refer to variations of the memory read opcode MemRd, MemRdData, MemRdTEE, MemRdDataTEE, or other memory read opcodes defined in CXL.mem. The first physical address may be associated with a first address space, such as the NPA space, and the second physical address may be associated with a second address space, such as the HPA space or SPA space, and the translation involves translating the first physical address to the second physical address. In addition, or alternatively, the first and second physical addresses may be associated with the same address space, such as the global address space, PGAS, pod address space, virtual pod address space, or fabric address space. Elements may communicate through one or more intermediary components, such as switches, retimers, or other appropriate entities that facilitate information transfer.

[0046] In some implementations of this technique, a UPLI request further includes the ReqSrcPhysAccID, ReqDstPhysAccID, ReqLen, ReqTag fields, a ReqAddr field containing the initial physical address, and a ReqCmd field containing the read command. Furthermore, it includes translating the ReqTag to a tag associated with the CXL.mem M2S request. The ReqSrcPhysAccID and ReqDstPhysAccID fields contain identifiers that the RPU uses to route the UPLI request to its target, and may also be used to construct response routing information. The ReqLen field may indicate a data transfer size of up to 256 bytes. If ReqLen indicates a transfer size exceeding the CXL.mem cache line size (e.g., 64 bytes), the RPU may translate the UPLI request into multiple CXL.mem M2S requests. Tag translation involves maintaining a bidirectional mapping between UPLI ReqTag values ​​and CXL.mem Tag values, enabling proper correlation between CXL.mem responses and corresponding UPLI requests.

[0047] In some implementations, this technique further includes receiving a sublosson-to-master data response (S2M DRS) of CXL.mem from the second entity. This process includes MemData, tags, and data; the CXL.mem S2M DRS is converted into a UPLI read response (RdRsp), consisting of the RdRspSrcPhysAccID field, RdRspDstPhysAccID field, RdRspTag field, and RdRspData; and the UPLI RdRsp is sent to the first entity. RdRspSrcPhysAccID corresponds to the ReqDstPhysAccID of the original UPLI request, and RdRspDstPhysAccID may correspond to a ReqSrcPhysAccID reflecting the routing path of the response. RdRspTag can be obtained from a bidirectional mapping maintained by the RPU, allowing the first entity to associate the response with the original request. In some cases, the RPU may also accumulate data from one or more CXL.mem S2M DRS messages before sending the data via the UPLI RdRsp, such as when a CXL.mem M2S request is generated from a UPLI request.

[0048] Some implementations of this method include a Poison field in CXL.mem S2M DRS and an RdRsp DataError field in UPLI RdRsp. Furthermore, it includes converting the Poison field in CXL.mem S2M DRS to the RdRsp DataError field in UPLI RdRsp. The Poison field in CXL.mem S2M DRS may indicate that the returned data contains errors. The RdRsp DataError field in UPLI may function as a per-beat data poison indicator. This conversion of error indications across protocol boundaries allows the initial entity to detect data corruption occurring in the CXL domain and take appropriate recovery actions, such as discarding corrupted data, retrying the request, or reporting the error to system management software.

[0049] In some implementations of this technique, a read command consists of a Read Class vendor-defined command, where the first entity is an accelerator or UALink switch, the second entity is a CXL device, and the second physical address is the host physical address (HPA) used by the second entity. Read Class VDCs support ReqCmd encoding and may enable vendor-specific memory access operations beyond standard UPLI read commands. CXL devices include CXL memory expanders, CXL memory pools, Global Fabric Attached Memory (G-FAM) devices (GFDs), or CXL accelerators. The HPA may represent an address in the address space used by the second entity to process memory requests.

[0050] In some implementations of this technique, a UPLI request indicates an I / O coherent read, and the CXL.mem M2S Req MemRd* further consists of a SnpType field containing SnpCur, a MetaField field containing Meta0-State(MS0), and a MetaValue field containing Invalid(I). The combination of SnpType(SnpCur), MetaField(MS0), and MetaValue(I) within a CXL.mem M2S request may indicate an intention to perform an I / O coherent read by requesting current data that is not cacheable. This combination may correspond to the I / O coherency model used in UALink, where a read from peer memory returns the most recent coherent copy from memory or cache in the destination system node. The RPU can select the values ​​of SnpType, MetaField, and MetaValue based on a predefined, predetermined, configurable, rule-based, or dynamic intent mapping between the UPLI I / O coherent read semantics and the CXL.mem coherency fields.

[0051] In some implementations, this technique may also involve sending a CXL.mem M2S request containing a MemSpecRd to a second entity. Speculative memory reads are initiated by the RPU to facilitate data availability from a second entity before the first entity explicitly requests that data or when it is unnecessary. The decision to initiate a speculative read may be based on pattern recognition algorithms that analyze the memory access behavior of the first entity, statistical models that predict future access locations, configurable prefetch policies that define aggression and speculation scope, or bandwidth availability assessments that determine whether the speculative operation will interfere with demand requests. When using the MemSpecRd opcode, some CXL.mem M2S Req fields such as Tag, MetaField, MetaValue, and SnpType may be reserved. Additionally, the RPU may issue a read (e.g., a CXL.mem M2S request containing a MemRd or MemRdData), prefetch data from the second entity, and buffer the returned data to satisfy the demand request from the first entity.

[0052] In some implementations of this technique, the first physical address is associated with the first address space, the second physical address is associated with a second address space different from the first, and the translation further translates the first physical address to the second physical address. The first address space may contain the NPA space used for UALink-based traffic, and the second address space may contain the HPA space used for CXL-based traffic. The first and second address spaces may differ in size, base address, memory layout, or granularity, but the translation can accommodate these differences while preserving the meaning of memory operations.

[0053] Some implementations of this method consist of instructions in a non-temporary computer-readable medium that, when executed by a processor, cause the processor to execute the method. Some implementations of this technique involve one or more integrated circuits configured to execute the technique, and these one or more integrated circuits constitute one of the following: (i) a general-purpose processing unit configured or connected via a communication interface (configured to execute the method by software and / or firmware execution); (ii) firmware and / or hardware logic circuits integrated into electronic equipment, which utilize hardware acceleration or special processing capabilities not typically provided by general-purpose processing units; or (iii) the operation of mounting one or more chiplets within one or more integrated circuit packages. Some implementations of this method involve an active cable consisting of first and second pluggable modules coupled in a physical medium, where the active cable further constitutes hardware circuitry integrated into the active cable and configured to execute the method. Some implementations of this technique involve a device acting as a switch, which consists of switching circuitry and is configured to execute the method.

[0054] In computing environments where external entities, such as accelerators, access memory resources connected to a processing unit, there may be scenarios where the processing unit provides access to memory resources through different memory paths. For example, a processing unit might have a first memory path from the UALink port to the first memory via the memory controller, and a second memory path from the UALink port to the second memory via the CXL port. The processing unit may include an UALink-based protocol such as UPLI and an RPU that performs translation between protocols for accessing the first and second memories. The RPU performs physical address translation from NPA to HPA, allowing the external entity to access both memory resources through the UALink port.

[0055] In various implementations, a system might consist of a processing unit comprising an Ultra Accelerator Link (UALink) port, a memory controller connected to the first memory, and a Compute Express Link (CXL) port coupled to the second memory, where the UALink port is configured to communicate with entities according to an UALink-based protocol. The processing unit is configured to provide the first memory path from the UALink port through the memory controller, and the second memory path from the UALink port to the second memory via the CXL port. Furthermore, the processing unit constitutes a Resource Provisioning Unit (RPU), configured to receive the first UALink protocol-level interface (UPLI) request from the entity, forward the first translation request to the first memory via the first memory path, receive the second UPLI request from the entity, and forward the second translation request to the second memory via the second memory path. The processing unit may be implemented as a processor, a system-on-a-chip (SoC), or a chiplet within an IC package. The first memory contains DRAM coupled to the memory controller, and the second memory contains a CXL memory expander, a CXL memory pool, or a CXL device exposing memory resources. The RPU performs physical address translation, determines whether a UPLI request targets the first or second memory location, and can route the translated request to the appropriate memory path. According to UPLI, entities include accelerators, CPUs, or switches that communicate with processing units via UALink ports. In some examples, the requested data may be served not from the first or second memory location, but from the processing unit's cache, such as the LLC.

[0056] In some system implementations, the processing unit is comprised of an even more coherent interconnect, with the first and second memory paths passing through a portion of this coherent interconnect. Coherent interconnects include mesh networks, ring interconnects, crossbars, network-on-chip (NoC), or other types of interconnect fabrics that maintain cache integrity between processing cores and other components of the processing unit. The first memory path might pass through a coherent interconnect from the RPU to the memory controller, and the second memory path might pass through a coherent interconnect from the RPU to the CXL port. In some examples, the RPU might be converted between UALink-based protocols such as UPLI and the protocols used by the coherent interconnect. Paths from the RPU to different memory locations may pass through other components connected to the coherent interconnect, such as cache / home agent (CHA) slices, snoop filter (SF), LLC slices, etc., which is optional for coherency resolution.

[0057] In some system implementations, the CXL port is connected to the CXL / PCIe root port (RP) and coherent interconnect. The CXL / PCIe RP is a separate component on the coherent interconnect, allowing the processing unit to communicate with the CXL device connected to the second memory. In other examples, the CXL / PCIe RP may be contained within the RPU.

[0058] In some system implementations, the processing unit further comprises a processing core, a cache / home agent (CHA), a snoop filter (SF), and a last-level cache (LLC) slice connected to a coherent interconnect. Additionally, it may include either a PCIe root port connected to an I / O device or an intersocket link (ISoL) port connected to a second processing unit. The processing core, CHA / SF / LLC slices, and additional ports are connected to a coherent interconnect, enabling coordinated access to memory resources. The PCIe RP may be coupled with I / O devices such as network controllers, Ethernet NICs, InfiniBand adapters, and PCIe GPUs. The ISoL port can utilize NVIDIA NVLink-C2C, ARM CHI C2C, or ICPIP for intersocket or inter-chip communication.

[0059] In some system implementations, the UALink port, CXL port, and memory controller are located within the same integrated circuit (IC) package. Furthermore, the RPU translates physical addresses associated with the UALink-based protocol to physical addresses associated with the processing unit, enabling entities to access both the first and second memory locations. The IC package may be implemented as a monolithic die or as a chiplet within a multi-chip module. Physical address translation may involve translating the network physical address (NPA) carried in the UPLI request to the host physical address (HPA) used in the processing unit's address space. The translated addresses are used by the processing unit to determine whether a particular request targets the first or second memory location and to route the translated request to the appropriate memory path.

[0060] In some implementations of the system, the second translation request consists of a CXL.mem Master-to-Subordinate (M2S) request, composed of MemRd*, and the CXL port is configured to send the CXL.mem M2S request to the second memory. Furthermore, the CXL port is configured to receive a CXL.mem lower-level data response (S2M DRS) to the master containing the memory data and data from the second memory, and the RPU is configured to translate the CXL.mem S2M DRS into a UPLI read response (RdRsp) containing the data, and send the UPLI RdRsp to the entity. The second memory path uses CXL.mem to communicate between the CXL port and the second memory, and the RPU performs address, tag, and opcode translation between UPLI and CXL.mem. The second memory contains CXL memory expanders and CXL devices corresponding to the CXL.mem S2M DRS message containing the requested data for the CXL.mem M2S request.

[0061] In computing environments where accelerator clusters are coupled via switches, accelerators may need to access memory resources outside the UALink domain. For example, memory resources such as CXL memory expanders, CXL memory pools, and GFDs may be coupled to the cluster via an RPU that translates the UALink-based protocol used by the accelerators to CXL.mem used by the CXL memory devices. In some configurations, the RPU may be connected to multiple different CXL memory devices and route translated requests to different CXL memory devices based on the physical address contained in the UPLI requests received from the accelerators. Thus, the RPU enables accelerators in the cluster to access a pool of CXL memory resources distributed across devices, and the switches provide the communication fabric between accelerators and between accelerators and the RPU.

[0062] Various implementations include a system with the following components: a switch; accelerators connected to the switch, communicating with each other according to an UALink-based protocol; a resource provisioning unit (RPU) connected to the switch; the first Compute Express Link (CXL) memory device connected to the RPU; and a second CXL memory device connected to the RPU, where the RPU is configured to receive the first UALink protocol-level interface (UPLI) request and the second UPLI request from the first accelerator via the switch, translate the first UPLI request into the first CXL.mem Master-to-Subordinate (M2S) request, send the first CXL.mem M2S request to the first CXL memory device, translate the second UPLI request into the second CXL.mem M2S request, and send the second CXL.mem M2S request to the second CXL memory device. This system allows accelerators within a UALink cluster to access CXL memory resources located outside the UALink domain without requiring changes to the accelerators' UALink interface or protocol. The RPU can determine which CXL memory device to target for each translation request based on the physical address included in the UPLI request. This can be done, for example, by comparing it with an address range register or a translation table that maps address ranges to specific CXL memory devices. The first and second CXL memory devices may differ in capacity, performance characteristics, address range, or device type. The RPU may be implemented as a discrete component connected to a switch, as an IP block embedded within an accelerator, or as a chiplet within an IC package. The RPU can perform translations between UPLI and CXL.mem, including translations of opcodes, addresses, tags, and additional fields.In some cases, the RPU may be coupled with two or more CXL memory devices, distributing translation requests across the CXL memory devices based on address, load balancing policy, or other criteria. This method can be implemented in hardware, firmware, software, or a combination of these.

[0063] In some system implementations, at least one of the first or second CXL memory devices consists of a Global Fabric Attached Memory Device (GFD). Furthermore, the RPU receives a Data Response from the CXL.mem sublogue to the master (S2M DRS) from the GFD, converts the CXL.mem S2M DRS into a UPLI Read Response (RdRsp), and is configured to send the UPLI RdRsp to the first accelerator via a switch. The GFD provides a large memory resource accessible via CXL.mem, which may also be shared between requesters via the RPU, with accelerators and hosts directly accessing CXL.mem. The RPU may convert the CXL.mem S2M DRS, converting the tag back to the original UPLI ReqTag, formatting the data as UPLI RdRspData, and delivering it to the first accelerator.

[0064] In some implementations, the system may consist of a host connected to at least one of the first CXL memory devices or a second CXL memory device via CXL.mem. Here, both the first accelerator (RPU) and the host access at least one of the first or second CXL memory devices. A shared access configuration allows both the accelerator and the host to access the same CXL memory resource, which can be used for data sharing, producer-consumer communication, or hierarchical memory management. The host can access the CXL memory device via CXL.mem without translation, while the accelerator accesses the same CXL memory device through the RPU, which performs translation between UPLI and CXL.mem.

[0065] In some implementations, the RPU is configured with a CXL fabric connecting it to a first and second CXL memory device, where at least one of the first or second CXL memory devices is either a CXL memory exer, a CXL memory pool, or a global fabric-attached memory device (GFD). The CXL fabric may include one or more CXL switches that provide connectivity between the RPU and the CXL memory devices. The CXL fabric may also allow the RPU to reach CXL memory devices that are not directly coupled to the RPU.

[0066] In some implementations of the system, the first UPLI request consists of the first physical address associated with the first address space, and the second UPLI request consists of the second physical address associated with the first address space. Here, the first CXL.mem M2S request consists of the third physical address associated with the second address space, and the second CXL.mem M2S request consists of the fourth physical address associated with the second address space. The RPU then translates the first physical address to the third physical address and the second physical address to the fourth physical address. The first address space consists of the Network Physical Address (NPA) space or the System Physical Address (SPA) space, and the second address space consists of the Host Physical Address (HPA) space. Address translation can be implemented using lookup tables, page tables, base and offset calculations, range-based mapping, or programmable translation functions. The RPU may determine which CXL memory device to target based on the translated addresses by comparing, for example, the third or fourth physical address with the address ranges assigned to the first and second CXL memory devices. In some cases, the first and second address spaces may be the same, such as the global address space or the fabric address space, in which case the RPU may perform routing without address translation.

[0067] In some implementations of the system, the switch consists of an UALink switch with a route table, which routes the first and second UPLI requests to the RPU based on the Destination Accelerator IDs contained in the first and second UPLI requests. The accelerator also communicates with the UALink switch via UPLI request channels and UPLI response channels. The UALink switch routes UPLI traffic based on the ReqDstPhysAccID field of each UPLI request and uses the route table to map the Destination Accelerator ID to an exit port linked to the RPU. Thus, the RPU is recognized by the accelerator as an UALink endpoint identified by the accelerator ID, allowing UPLI requests to be sent to the RPU using the standard UALink routing mechanism. The route table may be programmed by a pod controller or other management body. The UPLI request channel can carry read, write, atomic, and vendor-defined commands, and the UPLI response channel may carry the corresponding read and write responses.

[0068] In environments where entities communicating according to UPLI need to write data to memory resources combined with entities communicating according to CXL.mem, the RPU or other translation device may perform translations between UPLI write requests and CXL.mem write requests. The write path includes translating UPLI requests and originator data channels to CXL.mem M2S RwD messages, and translating CXL.mem S2M NDR completions to UPLI write responses (WrRsp). The RPU can translate opcodes, commands, addresses, tags, byte enablement, and completion states between the two protocol domains. In some examples, the RPU may split a UPLI write request with a transfer size exceeding the CXL.mem cache line size into multiple CXL.mem M2S RwD requests, aggregate the corresponding completions, and then return a UPLI WrRsp to the original entity.

[0069] In various implementations, there are methods for translating a write request from the Ultra Accelerator Link (UALink) protocol-level interface (UPLI) to a write request to Compute Express Link (CXL), which include: communication with the first entity based on UPLI; communication with another entity according to CXL.mem; receiving a UPLI request from the first entity consisting of a write command, the first physical address, and the write data; translating the UPLI request into a master-to-subordinate request (M2S RwD) in CXL.mem, including MemWr* and the second physical address; sending the CXL.mem M2S RwD and the write data to the second entity, receiving a Subordinate-to-Master No Data Response (S2M NDR) in CXL.mem from the second entity, and translating the CXL.mem S2M NDR into a UPLI write response (WrRsp); and sending the UPLI WrRsp to the first entity. Write translation allows entities communicating according to UPLI to potentially store data in memory resources associated with entities communicating according to CXL.mem. This method can be implemented in hardware, firmware, software, or a combination thereof, and may be performed by various devices such as processors, switches, bridges, RPUs, and semiconductor devices. The first physical address may be associated with a first address space, such as the NPA space, and the second physical address may be associated with a second address space, such as the HPA space, where the translation involves translating the first physical address to the second physical address. Furthermore, or alternatively, the first and second physical addresses may be associated with the same address space, such as the global address space, PGAS, pod address space, virtual pod address space, or fabric address space. UPLI write commands may include Write, WriteFull, or Write Class Vendor Defined Commands as defined in the UPLI specification.

[0070] In some implementations of this technique, a UPLI request further consists of the ReqSrcPhysAccID field, the ReqDstPhysAccID field, the ReqTag field, and the ReqAddr field, which constitutes the initial physical address. The write data is received in a UPLI originator data (OrigData) channel containing OrigDataByteEn. This also involves converting the ReqTag to a tag associated with the CXL.mem M2S RwD. The ReqSrcPhysAccID and ReqDstPhysAccID fields may contain identifiers used to construct routing and response routing information for the UPLI request. The OrigDataByteEn field may contain per-byte emblem bits indicating the valid bytes of the write data. Tag conversion involves maintaining a bidirectional mapping between UPLI ReqTag values ​​and CXL.mem Tag values, enabling proper correlation between CXL.mem S2M NDR completion and the corresponding UPLI write request.

[0071] In some implementations of this method, MemWr* is composed of MemWrPtl, and the write data consists of a partial cache line update. The bytes associated with the UPLI originator's data channel are also utilized to indicate which bytes in the cache line will be written by the second entity. The MemWrPtl opcode may indicate a partial write where only a portion of the bytes in the CXL.mem cache line are updated. Byte enablement from the UPLI OrigDataByteEn field can propagate to the CXL.mem domain, allowing the second entity to update only the specified bytes and preserve the rest of the cache line.

[0072] In some implementations of this method, UPLI WrRsp further includes WrRspTag and WrRspStatus fields, and CXL.mem S2M NDR further consists of Cmp* complementary opcodes. Furthermore, it also includes converting the tags of CXL.mem S2M NDR to WrRsp tags in UPLI WrRsp, and converting the completion status of CXL.mem S2M NDR to WrRspStatus in UPLI WrRsp. WrRspTag can be obtained from a bidirectional mapping maintained by the RPU, allowing the initial entity to associate the write response with the original write request. WrRspStatus may indicate the success or failure of the write operation. The conversion of completion status across protocol boundaries allows the initial entity to detect write failures that occurred in the CXL domain and take appropriate recovery actions.

[0073] In some implementations of this technique, the first entity consists of an accelerator, and the second entity consists of a CXL device, which includes either a CXL memory expander, a CXL memory pool, or a global fabric-attached memory device (GFD). The first physical address is associated with the network physical address (NPA) space, and the second physical address is associated with the host physical address (HPA) space. Here, translation further means translating the first physical address to the second physical address. Address translation from NPA to HPA may be implemented using lookup tables, page tables, base and offset calculations, range-based mapping, or programmable translation functions. The GFD provides a large memory resource accessible via CXL.mem and may be shared among requesters.

[0074] In some implementations of this method, the UPLI request further includes a ReqLen field indicating a transfer size exceeding the CXL.mem cache line size. Furthermore, the translation generates CXL.mem M2S RwD requests from the UPLI request, each containing its respective MemWr* and write data portion. While a UPLI write request may have a ReqLen indicating a transfer size of up to 256 bytes, a CXL.mem M2S RwD message may carry up to 64 bytes of data per request. If the ReqLen exceeds the CXL.mem cache line size, the RPU splits the UPLI write request into CXL.mem M2S RwD requests, each conveying its write data portion with its own translated address. The RPU aggregates the corresponding CXL.mem S2M NDR completions and may then return the UPLI WrRsp to the original entity.

[0075] Figure 4A shows an example of a system consisting of a memory-coupled RPU, which allows external entities to access resources coupled to the RPU. The RPU may be converted between UALink-based protocols (such as UPLI) and CXL-based protocols (such as CXL.mem). Furthermore, the RPU may be converted between UPLI and CXL.io, or between UPLI and CXL.cache. In some examples, the RPU may be implemented as a separate component on a PCB and coupled with other components such as a CPU, GPU, accelerator, switch, or CXL device. In other examples, the RPU may be integrated into other silicon designs, such as IP within a processor, or implemented as a chiplet within an IC package. The RPU is coupled to an initial entity (Entity.1), which may be an accelerator, GPU, CPU, switch, originator, or consumer, and the RPU can communicate with the initial entity according to a UALink-based protocol such as UPLI. The RPU is further coupled with a second entity (Entity 2), which is either CXL memory, a CXL device, a switch, or a provider, and the RPU can communicate with the second entity according to a CXL-based protocol such as CXL.mem, CXL.io, or CXL.cache. In some examples, UALink-based traffic, such as UPLI traffic, may be associated with a first address space, such as the NPA space, while CXL-based traffic, such as CXL.mem traffic, may be associated with a second address space, such as the System Physical Address (SPA) space or the Host Physical Address (HPA) space. In this case, the RPU can perform address translation between the first and second address spaces, such as between addresses in the NPA space, within the SPA space, or between addresses in the HPA space.In other examples, UALink-based and CXL-based traffic may be associated with the same physical address space, such as the global address space, partitioned global address space (PGAS), pod address space, virtual pod address space, or fabric address space. In this case, the RPU can perform address translation between addresses within the same address space. The RPU may also perform translations such as opcodes, commands, and TLP translations, for example, translating commands in a request compliant with the UALink-based protocol (e.g., a UPLI vendor-defined read command) to opcodes in a request compliant with the CXL-based protocol (e.g., CXL.mem MemRd). The RPU can further perform translations between messages compliant with the UALink-based protocol and messages compliant with the CXL-based protocol, tag translations, traffic class (TC) translations, and cross-field translations between CXL.mem Tags and UPLI ReqTags, or between UPLI RdRspTags and CXL.mem Tags. In addition, the RPU may maintain tracking between tags in the UPLI domain and tags in the CXL domain, associating responses with corresponding requests. RPUs can also translate mislabelings such as those for poison.

[0076] Figure 4B shows an example of translation performed by the RPU on TFD. This is a translation between UPLI traffic used for communication with the first entity (Entity.1, accelerator, GPU, CPU, switch, originator, consumer) and CXL.mem traffic used for communication with a second entity (Entity.2), such as CXL devices, CXL memory, and CXL switches. Furthermore, or alternatively, the RPU may perform translations between UPLI and CXL.io requests, or between UPLI and CXL.cache requests. The first entity may initiate a UPLI transaction (Req) containing a UPLI request (Req) which includes a Request Command (e.g., ReqCmd(Read)), Request Source Physical Accelerator ID (e.g., ReqSrcPhysAccID(a.1)), Request Destination Physical Accelerator ID (e.g., ReqDstPhysAccID(b.1)), Request Tag (e.g., ReqTag(p.2.1)), and Request Address (e.g., ReqAddr(AS.2.1)). The RPU may then convert the UPLI transaction into a CXL.mem transaction, which may include a CXL.mem M2S request containing a MemOpcode (MemRd*), Tag (p.1.1), and Address (AS.1.1), and may also send the CXL.mem M2S request to a second entity. The asterisk in a translated CXL.mem M2S request indicates that it may represent a suitable superset combination of read opcodes, commands, and operations supported by CXL.mem (e.g., MemRd, MemRdData, MemRdTEE, MemRdDataTEE). The RPU may also play a role in converting between other fields in the UPLI transaction and fields in the CXL.mem transaction, such as address fields, tag fields, QoS-related fields, and identification (ID) fields, and routing the UPLI request to its target.

[0077] Upon receiving a response from the second entity, if it includes a CXL.mem S2M DRS containing Opcode (MemData), Tag (p.1.1), and Data (*Data.1*), the RPU converts the CXL.mem S2M DRS into a UPLI read response / data (RdRsp), which includes the Read Response Source physical accelerator ID (e.g., RdRspSrcPhysAccID(b.1)), Read Response Destination physical accelerator ID (e.g., RdRspDstPhysAccID(a.1)), response transaction tag (e.g., RdRspTag(p.2.1)), and response data (e.g., RdRspData(*Data.1*)). Optionally, the RPU can also act as an endpoint or a completion device, terminating the UPLI transaction. The RPU issues CXL.mem transactions and acts as an independent protocol initiator, such as a CXL host, and may also utilize the translation fields of UPLI transactions to construct CXL.mem transactions. The RPU may also perform further operation code and command translations, such as the translation between vendor-defined read commands in OPLI requests and MemRd in CXL.mem requests. In addition, the RPU translates between UPLI-compliant and CXL.mem-compliant messages, translates tags, and translates error indicators such as poison.

[0078] In some cases, the RPU may convert a UPLI transaction into multiple CXL.mem transactions, for example, a UPLI request containing a request length field such as ReqLen, which contains a value to read up to 256 bytes of data. In this case, the RPU may convert the UPLI request into a CXL.mem M2S request, each possibly representing up to 64 bytes of data as a cache line. The RPU may also convert between CXL.mem responses such as CXL.mem S2M NDR and CXL.mem S2M DRS and UPLI responses (UPLI read responses), transferring the read data contained in the CXL.mem DRS message to the UPLI read response. In some cases, the RPU may accumulate data from one or more CXL.mem DRS messages and then send the data in a UPLI read response.

[0079] Figure 5A shows an example of a system consisting of an RPU (Processor, Accelerator, Switch, etc.) where external entities can access resources (such as CXL devices and CXL memory) coupled to the RPU. The RPU is coupled to a first entity (Entity.1), which is an accelerator, GPU, CPU, UALink switch, or consumer, and the RPU can communicate with the first entity according to a UALink-based protocol such as UPLI. The RPU is further coupled to a second entity (Entity.2), which is a CXL device, CXL memory, CXL-based memory pool, CXL switch, MxPU, or provider, and the RPU communicates with the second entity according to a CXL-based protocol such as CXL.mem. In some examples, a UALink-based protocol such as UPLI may be associated with a first address space such as the NPA space, and a CXL-based protocol such as CXL.mem may be associated with a second address space such as the HPA space. In this case, the RPU can perform address translation between the first and second addresses, such as between addresses in the NPA space or between addresses in the HPA space. In other examples, UALink-based protocols like UPLI and CXL-based protocols like CXL.mem may be associated with the same physical address space, such as the global address space. In this case, the RPU can perform address translation between addresses within the same address space. The RPU may also perform further translations, such as opcode and command translations, including the translation between the Read command in an OPLI request and the MemRd in a CXL.mem request. Furthermore, the RPU also translates between UPLI-compliant and CXL.mem-compliant messages, translates tags, and translates error indicators such as poison.

[0080] Figure 5B shows an example TFD illustrating RPUs that can be translated between UALink-based traffic (such as UPLI traffic) and CXL-based traffic (such as CXL.mem traffic). Furthermore, RPUs can be translated between UPLI and CXL.io, or between UPLI and CXL.cache. RPUs can also provide intent-based translation between protocols, such as between UPLI and CXL.mem, identifying the intent of an incoming transaction and generating a translation transaction that conveys the corresponding intent. They can also convey intents based on predefined, predetermined, configurable, rule-based, or dynamic mapping between intents. The RPU can receive UALink UPLI transactions from the first entity (Entity.1), for example, an accelerator, which may include a UPLI request containing a Request Command (e.g., ReqCmd(Read)), Request Source Physical Accelerator ID (e.g., ReqSrcPhysAccID(a.1)), Request Destination Physical Accelerator ID (e.g., ReqDstPhysAccID(b.1)), Request Address (e.g., ReqAddr(AS.1.1)), Request Tag (e.g., ReqTag(c.1.1)), Request Length (e.g., ReqLen(d.1.1)). The received UPLI transaction may indicate an intention to perform an I / O coherent read, for example, a request to UALink for a copy of the latest data corresponding to a typical I / O coherence model.

[0081] The RPU can convert UPLI transactions into CXL.mem transactions, which include CXL.mem M2S requests containing memory operations (e.g., MemOpcode(MemRd)), snoop types (e.g., SnpType(SnpCur)), metadata fields (e.g., MetaField(MS0)), metadata values ​​(e.g., MetaValue(I)), tags (p.2.1), and addresses (AS.2.1)). This conversion from UPLI to CXL.mem may indicate an intention to perform I / O coherent reads through CXL.mem requests seeking non-cacheable current values. In this case, it may be represented as a 64B cache line corresponding to the request length in the UPLI request (e.g., ReqLen(d.1.1)). The RPU may further convert between other values ​​in the UPLI transaction and the CXL.mem transaction, such as between addresses, tags, QoS-related values, and identification numbers (IDs), and may play a role in routing the UPLI request to its destination.

[0082] In some cases, the RPU may convert a UPLI transaction into multiple CXL.mem transactions. For example, if a UPLI request consists of a request length field (e.g., ReqLen) with a value indicating a read of 64 bytes or more of data, the RPU may convert that UPLI request into a CXL.mem M2S request, each potentially containing up to 64 bytes of data. This could possibly represent a 64-byte cache line. The RPU may also convert between CXL.mem responses such as CXL.mem S2M NDR and CXL.mem S2M DRS and UPLI responses (UPLI read responses), transferring the read data contained in the CXL.mem DRS message via the UPLI read response.

[0083] In some cases, the RPU may receive a response from a second entity (Entity 2), which may include a CXL.mem S2M NDR containing Opcode (Cmp) and Tag (p.2.1), and possibly a CXL.mem S2M DRS containing Opcode (MemData), Poison (E), Tag (p.2.1), and Data (*Data*). The RPU can then convert the CXL.mem S2M DRS into a UPLI read response / data (RdRsp data) which includes the read source physical accelerator ID (e.g., RdRspSrcPhysAccID(b.1)), read response destination physical accelerator ID (e.g., RdRspDstPhysAccID(a.1)), read response transaction tag (e.g., RdRspTag(c.1.1)), read response data error (e.g., RdRspDataError(e)), and read response data (e.g., RdRspData(*Data*))). This conversion demonstrates that the RPU can propagate error responses from the CXL domain to the UPLI domain. For example, it can convert an error indication (poison) contained in a CXL.mem S2M DRS message to an error indication contained in a UPLI RdRsp message (e.g., Read Response Data Error (e.g., RdRspDataError)). Furthermore, the RPU may accumulate data from one or more CXL.mem DRS messages before sending the data in a UPLI read response.

[0084] Figure 6A shows an example of a system including a processor, allowing external entities such as GPUs and accelerators to access memory resources mapped to the address space used by the coherent interconnect. For example, access is possible through one or more of the two paths indicated as (E.1)-(M.1) and (E.2)-(M.2). The processor includes processing cores, a cache / home agent (CHA), a snoop filter (SF), and an LLC, which may be implemented coupled to the coherent interconnect as distributed slices. The processor may also include PCIe RP, CXL / PCIe RP that can work with network controllers such as Ethernet NICs and InfiniBand adapters, a memory controller that can connect to the first memory (Memory.1, DRAM, etc.), an ISoL port using NVIDIA NVLink-C2C, ARM CHI C2C, or the Intel Coherent Processor Interconnect Protocol (ICPIP) such as Intel UPI. The processor may be coupled to a second memory (Memory.2), such as a CXL memory expander, and may also include or be coupled to an RPU that communicates with entities according to an UALink-based protocol (such as UPLI) in conjunction with the UALink port. In this port, the RPU performs physical address translation, allowing entities to access the first memory, for example, over path (E.1)-(M.1) or path (E.2)-(M.2) to access the second memory. The illustrated RPU may be coupled to a coherent interconnect and may translate between the UALink-based protocol and the protocol used by the coherent interconnect. The processor may be implemented as an IP block embedded in silicon designs such as switches and accelerators.In other examples, processors may be implemented as monolithic dies, as chiplets within IC packages, or as components on a substrate, utilizing mesh-based coherent interconnects, rings, crossbars, network-on-chip (NoC), or other coherent interconnects.

[0085] Figure 6B shows an example of two UPLI requests (such as UPLI read requests) that are received by a TFD from an entity such as a GPU or accelerator, processed by the RPU, and forwarded to different memory locations using the protocol used in the coherent interconnect. The path from the RPU to the different memory locations may also pass through CHA / SF / LLC slices, memory controllers, or, in other examples, home agents or home nodes, which is done optionally for consistency resolution. The RPU may perform physical address translation, such as from network physical addresses (NPAs) to host physical addresses (HPAs), to allow entities to access the processor's memory. A processor may have multiple memory resources. For example, first memory (Memory.1) may be coupled with the processor's memory controller and DRAM, and second memory (Memory.2) may be coupled with a CXL memory expander and the processor's CXL / PCIe RP. The RPU may also perform additional translations, such as protocol translation from UALink-based protocols like UPLI to protocols used by the coherent interconnect, and optionally send the translated request to the coherent interconnect to request a read from memory. In some cases, the requested data may be provided not from memory, but from a processor cache such as the LLC. The data is returned to the RPU through the coherent interconnect, and the RPU provides a UPLI read response / data (RdRsp) to the requesting entity.

[0086] The TFD shows two exemplary transactions between an entity and the RPU, each corresponding to a different physical address mapped to a different memory resource. The first exemplary transaction includes an initial UPLI request (Req) containing a physical address (AS.2.1), which may also be an NPA. The RPU forwards this to the first memory (Memory.1) via the Coherent Interconnect Protocol and the memory controller, retrieves *Data.1*, and sends it to the entity using the first UPLI RdRsp via the Coherent Interconnect Protocol and the RPU. The second exemplary transaction includes a second UPLI request containing a physical address (AS.2.2), which may also be an NPA. The RPU translates this to a physical address (AS.1.2) and forwards it to the second memory (Memory.2) using a CXL.mem M2S request via the Coherent Interconnect Protocol and the CXL / PCIe RP. *Data.2* is retrieved from the second memory location using CXL.mem S2M DRS and sent to the RPU via the Coherent Interconnect Protocol. The RPU can then send *Data.2* through a second UPLI RdRsp. The physical addresses (AS.2.1) and (AS.2.2) may refer to different memory regions within the NPA address space exposed through the UALink port, allowing the RPU to access memory resources based on its translation capabilities.

[0087] Figure 7A shows an example of a system including a processor, which allows external entities such as GPUs and accelerators to access memory resources mapped to the address space used by the coherent connection via one of two illustrated paths, indicated as (E.1)-(M.1) and (E.2)-(M.2). The processor includes processing cores, a cache / home agent (CHA), a snoop filter (SF), and an LLC, which may be implemented coupled to the coherent interconnect as distributed slices. The processor may further include a PCIe RP that can be coupled with a PCIe GPU, a memory controller (such as DRAM) that interacts with the first memory (Memory.1), and ISoL ports such as NVIDIA NVLink-C2C, arm chi C2C, and ports utilizing the Intel Coherent Processor Interconnect Protocol (ICPIP, Intel UPI, etc.). The processor may include an UALink port or an RPU coupled with an UALink port, which can communicate with entities according to an UALink-based protocol such as UPLI. In this case, the RPU further includes a CXL RP coupled with a second memory (Memory.2), such as a CXL memory expander. The RPU performs physical address translation, allowing entities to access the first memory (e.g., on path (E.1)-(M.1)) or the second memory (e.g., on path (E.2)-(M.2)). The illustrated RPU is coupled with a coherent interconnect and may translate between UALink-based protocols and the protocols used by the coherent interconnect. The processor may utilize a mesh-based coherent interconnect, or it may utilize a ring, crossbar, network-on-chip (NoC), or other coherent interconnect.

[0088] Figure 7B shows an example of two UPLI requests (such as UPLI read requests) that are received by a TFD from an entity such as a GPU or accelerator, processed by the RPU, and forwarded to different memory locations using the protocol used in the processor's coherent interconnect. The path from the RPU to the different memory locations may also pass through CHA / SF / LLC slices, memory controllers, or, in other examples, home agents or home nodes, which is optional for consistency resolution. The RPU can perform physical address translation to enable access to the processor's memory resources, such as translating network physical addresses (NPAs) to host physical addresses (HPAs) and from NPAs to system physical addresses (SPAs). A processor may have multiple memory resources. For example, the first memory (Memory.1) may be the processor's memory controller and DRAM coupled together, and the second memory (Memory.2) may be a CXL memory expander coupled to the processor's CXL RP, which may be integrated into the RPU. The RPU may also perform additional translations, such as protocol translation, between UALink-based protocols like UPLI and protocols used in the coherent interconnect. The RPU optionally sends the translated UPLI request to the coherent interconnect, requesting a read from the first or second memory location. The RPU can further translate UALink-based traffic, such as UPLI traffic, in combination with traffic from CXL.mem, CXL.io, or CXL.cache, and the RPU can optionally send the translated UPLI traffic to the second memory location via the CXL RP. In some examples, the requested data may be served from a processor cache, such as the LLC, rather than from memory. The data is returned to the RPU through the coherent interconnect, and the RPU provides a UPLI read response / data (RdRsp) to the requesting entity.

[0089] The TFD shows two exemplary transactions between an entity and an RPU, each corresponding to a different physical address mapped to a different memory resource. The first exemplary transaction corresponds to a memory read path (E.1)-(M.1) and may contain an initial UPLI request (Req) containing the physical address (AS.2.1). This may also be an NPA, which the RPU forwards to the first memory via the Coherent Interconnect Protocol and the memory controller, resulting in *Data.1*, which is sent to the entity via the Coherent Interconnect Protocol and the RPU with the first UPLI RdRsp. The second exemplary transaction corresponds to a memory read path (E.2)-(M.2) and may contain a second UPLI request containing the physical address (AS.2.2). This may also be an NPA. The RPU translates the second UPLI request into a CXL.mem M2S request (MemRd* and Address (AS.1.2)), and the RPU can send the translated request to the second memory via the CXL RP. *Data.2* is retrieved from the second memory using the CXL.mem S2M DRS and sent to the RPU via the CXL RP. From there, the RPU can send *Data.2* to the entity using the second UPLI RdRsp.

[0090] Figure 8A shows an example of a system composed of computers, which includes switches and bridges and consists of a first interface (Interface.1) and a second interface (Interface.2). The first interface may communicate with the first entity (Entity.1), such as a CPU or accelerator, following an UALink-based protocol like UPLI. The second interface may communicate with the second entity (Entity.2), such as a switch, CXL memory expander, CXL memory pool, GFD, or CXL accelerator, following CXL.mem. The computer can extract addresses from requests received through the first interface, and these addresses may refer to the first address space, such as the network physical address (NPA) space used by the first entity. The computer can further translate these addresses and generate requests with the translated addresses to send via the second interface. These translated addresses may refer to the second address space used by the second entity. In other examples, the first and second address spaces may be associated with the same address space, such as a common address space, global address space, pod address space, or fabric address space, and the computer may perform address translation between addresses within the same common address space. This computer may also be implemented in an IC package with high-speed differential I / O balls positioned according to a ball grid array layout defined in the retimer specification.

[0091] Figure 8B shows an example of translations performed by the computer in TFD, such as address translation. The following translations are performed: (1) when the first physical address (such as the network physical address (NPA)) is received from the first entity (Entity.1, CPU or accelerator) in an UALink-based request (such as a UPLI request); and (2) when the second physical address (such as the host physical address (HPA)) is carried in a CXL request (such as a CXL.mem request) and sent to the second entity (Entity.2), which is a switch or CXL device. This allows the first entity to access resources mapped to the address space used by the second entity. The first entity can initiate a UPLI transaction, which includes ReqCmd (Read), ReqAddr (AS.3.1), and ReqTag (c.3.1). The computer converts the UPLI transaction into a CXL transaction, which may include a CXL.mem M2S request (MemOpcode(MemRd*), Tag(p.2.1), Address(AS.2.1)), and can send the CXL.mem M2S request to a second entity. Upon receiving one or more responses from the second entity, which may include a CXL.mem S2M DRS containing Opcode(MemData), Tag(p.2.1), and Data(*Data.1*), the computer converts one or more of these responses. For example, it converts the CXL.mem S2M DRS into a UPLI read response / data (RdRsp) consisting of RdRspTag(c.3.1) and RdRspData(*Data.1*), and sends the UPLI RdRsp to the first entity.

[0092] The computer can then initiate a predictive memory read targeting a second entity, which may include a CXL.mem M2S request containing a MemOpcode (MemSpecRd) and Address (AS.2.2). The computer may use the predictive memory read on behalf of the first entity to facilitate data prefetching and reduce read delays from the second entity. When using the MemSpecRd opcode, some CXL.mem M2S Req fields such as Tag, MetaField, MetaValue, and SnpType may be reserved. In addition, it may perform protocol conversions, opcode conversions, command conversions, TLP type conversions, and conversions between UALink-based domains and CXL domains. In some examples, the computer may issue multiple CXL.mem reads when it receives a UPLI request from the first entity. For example, the computer may issue CXL.mem M2S requests from MemRd or MemRdData. For example, this can be used when splitting a large data block (e.g., 256B) in a UPLI request into smaller CXL.mem reads (e.g., 64B), or when prefetching data from a second entity using a CXL.mem read. A computer can convert requests and transactions initiated from an UALink-based domain to a CXL domain, or from a CXL domain to an UALink-based domain, or convert requests and transactions initiated from both domains.

[0093] In environments requiring coordinated access to shared resources while using different protocols, the first entity communicating according to CXL.mem (e.g., a CXL host) may need to access memory resources residing within the UALink domain, such as memory connected to an accelerator or memory accessible via the UALink network. RPUs and other appropriate devices perform translations between CXL.mem and UPLI, facilitating memory operations, data transfer, and resource sharing across different protocol domains while maintaining the requirements of each protocol. These translations include translating CXL.mem M2S requests to UPLI requests and UPLI responses to CXL.mem S2M responses, encompassing the translation of opcodes, commands, addresses, tags, and additional fields. RPUs may also perform address translation between the host physical address (HPA) space used by CXL-based traffic and the network physical address (NPA) space used by UALink-based traffic, or between addresses within the same address space. RPUs include a cache for storing fetch data and may operate as a CXL type 2 device, a CXL type 3 device, or simultaneously expose both HDM-D and HDM-H memory regions.

[0094] In various implementations, there are methods for translating requests from Compute Express Link (CXL) requests to the Ultra Accelerator Link (UALink) protocol-level interface (UPLI), which include: communicating with the first entity based on CXL.mem; communicating with the second entity according to UPLI; receiving a CXL.mem Master-to-Subordinate (M2S) request from the first entity containing MemRd* and the first physical address; translating the CXL.mem M2S request into a UPLI request containing a read command and the second physical address; sending the UPLI request to the second entity; receiving a UPLI read response (RdRsp) containing data from the second entity, translating the UPLI RdRsp into a CXL.mem sub-to-master data response (S2M DRS), and sending the CXL.mem S2M DRS to the first entity. This conversion allows entities communicating according to CXL.mem to potentially access entities communicating according to UPLI (e.g., HBM or High Bandwidth Flash (HBF) connected to an accelerator, or memory accessible via the UALink network). This method can be implemented in hardware, firmware, software, or a combination thereof, and can be performed by various devices such as processors, switches, bridges, RPUs, and semiconductor devices. The first physical address may be associated with a first address space, such as the HPA space, and the second physical address may be associated with a second address space, such as the NPA space, in which case the first physical address may be translated to the second physical address. Furthermore, or alternatively, the first and second physical addresses may be associated with the same address space, such as the global address space, partitioned global address space (PGAS), pod address space, virtual pod address space, or fabric address space.

[0095] In some implementations of this technique, a CXL.mem M2S request further consists of a MemOpcode field (MemRd*), a tag field, and an Address field (the first physical address). Furthermore, a UPLI request includes a ReqCmd field containing the read command, a ReqAddr field containing the second physical address, a ReqSrcPhysAccID field, a ReqDstPhysAccID field, a ReqLen field, and a ReqTag field. It also includes converting tags to ReqTag. Tag conversion involves maintaining a bidirectional mapping between CXL.mem Tag values ​​and UPLI ReqTag values, enabling proper correlation between UPLI responses and corresponding CXL.mem requests. The ReqSrcPhysAccID and ReqDstPhysAccID fields may contain identifiers for routing the UPLI request to the target accelerator. If the transfer size exceeds the CXL.mem cache line size (e.g., 64 bytes), CXL.mem M2S requests may be merged into UPLI requests, which support a maximum transfer size of 256 bytes.

[0096] In some implementations of this technique, the UPLI RdRsp is further composed of data consisting of the RdRspSrcPhysAccID field, the RdRspDstPhysAccID field, the RdRspTag field, and the RdRspData field. The CXL.mem S2M DRS is further composed of an opcode field consisting of MemData and a tag. The RdRspDstPhysAccID corresponds to the ReqSrcPhysAccID of the original UPLI request and may reflect the routing path of the response. The RdRspTag is used to retrieve the corresponding CXL.mem tag from a bidirectional mapping maintained during request translation. In some examples, the UPLI RdRsp may carry data across multiple beats, accumulating the beats before the RPU constructs the CXL.mem S2M DRS.

[0097] In some implementations of this method, UPLI RdRsp further constitutes the RdRspDataError field, and CXL.mem S2M DRS further constitutes the Poison field. Furthermore, it also involves converting the RdRspDataError field of UPLI RdRsp to the Poison field of CXL.mem S2M DRS. The RdRspDataError field of UPLI may function as a per-beat data poison indicator. The Poison field of CXL.mem S2M DRS may indicate that the returned data contains errors. This cross-protocol conversion of error indications allows the initial entity to detect data corruption that occurred in the UALink domain and take appropriate recovery measures.

[0098] In some implementations, this technique further stores the UPLI RdRsp data in a cache and sends a No Data Response (S2M NDR) from CXL.mem to the master to the first entity, so that Cmp-S indicates the shared cache line state. By storing the retrieved data in the cache, the RPU can maintain a local copy of the cache line and participate in consistency management with the first entity. The S2M NDR that makes up Cmp-S notifies the first entity that the RPU holds a shared copy, so that both the first entity and the RPU can hold a cache copy simultaneously. This behavior may correspond to the semantics of CXL type 2 devices with an HDM-D memory region where the Device Coherency Engine (DCOH) manages the cache line state on behalf of the device.

[0099] In some implementations of this technique, a CXL.mem M2S request further consists of a SnpType field (SnpData), a MetaField field (Meta0-State, MS0), and a MetaValue field (Shared(S)). The CXL.mem M2S request targets the Host Managed Device Memory (HDM-D) memory region. The combination of SnpType(SnpData), MetaField(MS0), and MetaValue(S) may indicate the intention to first obtain a shared copy of the cache line. The RPU uses these fields to determine coherency behavior and select the Cmp-S opcode for S2M NDR.

[0100] In some implementations of this technique, the CXL.mem M2S request further includes a SnpType field containing No-Ops and a MetaField field containing No-Ops. When a CXL.mem S2M DRS is sent to the first entity, the first entity does not receive a CXL.mem Subordinate-to-Master No Data Response (S2M NDR). The CXL.mem M2S request targets the Host Managed Device Memory (HDM-H) memory region. The No-Ops values ​​in SnpType and MetaField may indicate that no snooping or metadata operations corresponding to CXL Type-3 device semantics are involved. In this flow, the RPU acts as a pass-through, transferring data from the UALink domain to the first entity without maintaining a cache copy or engaging in coherency management.

[0101] In some implementations, this technique further involves sending a CXL.mem Subordinate-to-Master No Data Response (S2M NDR) containing Cmp-S to the initial entity without caching the data. The RPU may respond with Cmp-S even without a cache or data storage. This is useful when the initial entity does not support CXL Type 3 device flows and only supports CXL Type 2 device flows. By emulating Type 2 behavior, compatibility with hosts that do not support Type 3 can be maintained.

[0102] In some implementations of this method, the first physical address is associated with a first address space, the second physical address is associated with a second address space different from the first, and the translation further translates the first physical address to the second physical address. The first address space consists of a host physical address (HPA) space or a system physical address (SPA) space, and the second address space consists of a network physical address (NPA) space. Address translation can be implemented using lookup tables, page tables, base and offset calculations, range-based mapping, or programmable translation functions. HPAs or SPAs can represent addresses within a CXL domain, and NPAs can represent addresses within an UALink network spanning multiple accelerators or nodes. In some examples, the first and second address spaces are the same address space, and the RPU may perform routing without address translation.

[0103] In some implementations of this method, the first entity is a CXL host, the second entity is an accelerator or UALink switch, and the read command is a Read command or Read Class Vendor Defined Command. The Read Class VDC may support ReqCmd encoding, which enables vendor-specific memory access operations beyond standard UPLI read commands. The CXL host may include a CPU or other CXL-enabled entities. The accelerator may include a GPU, TPU, or other processing device connected to the UALink network.

[0104] In some implementations, this technique further involves receiving a Master-to-Dependent Request (M2S RwD) for CXL.mem from the first entity, which includes MemWr* and data containing the write data; translating the CXL.mem M2S RwD into a UPLI request containing the write command; sending the UPLI request and write data to the second entity; receiving a UPLI Write Response (WrRsp) from the second entity; translating the UPLI WrRsp into a Sublog No Data Response (S2M NDR) from CXL.mem to the Master; and sending the CXL.mem S2M NDR to the first entity. MemWr* may refer to MemWr, MemWrPtl, MemWrTEE, MemWrPtlTEE, or other memory write opcode variants defined or defined in CXL.mem. Write data can be transferred from the CXL.mem domain to the UPLI domain via the Originator Data Channel, and byte validation operations using OrigDataByteEn are optionally possible when MemWr* refers to MemWrPtl during partial writing. UPLI write commands can include Write or WriteFull commands as defined in the UPLI specification. UPLI WrRsp may include WrRspTag and WrRspStatus to indicate the completion status of the write operation.

[0105] In some implementations, this technique also involves receiving a CXL.mem M2S request from the first entity, which is a MemSpecRd; and the CXL.mem M2S request containing the MemSpecRd may also be translated into a UPLI request. The MemSpecRd opcode indicates a presumptive read issued from the first entity before coherence resolution, and the RPU may translate it into a UPLI read request directed to the second entity. When using the MemSpecRd opcode, some CXL.mem M2S Req fields such as MetaField, MetaValue, and SnpType may be reserved. Furthermore, based on access pattern recognition and configurable prefetch policies, the RPU may initiate its own presumptive UPLI request to prefetch data from the second entity.

[0106] Some implementations of this method are performed by a device that exposes a first memory region using a CXL Type 2 device memory flow associated with a host-managed device memory-device coherent (HDM-D) region, and a second memory region using a CXL Type 3 device memory flow associated with a host-managed device memory-host coherent (HDM-H) region. The CXL specification allows a device to expose both HDM-D and HDM-H memory regions simultaneously. For example, local memory such as HBM or HBF may be connected to the accelerator and exposed as an HDM-D region using CXL Type 2 memory flows and coherency management. On the other hand, remote memory accessible via the UALink network may be exposed as an HDM-H region using CXL Type 3 memory flows. The RPU can select the appropriate memory flow based on the address of the CXL.mem M2S request.

[0107] Some implementations of this method consist of instructions in a non-temporary, computer-readable medium that, when executed by a processor, cause the processor to execute the method. Some implementations of this technique involve one or more integrated circuits configured to execute the technique, and these one or more integrated circuits constitute one of the following: (i) a general-purpose processing unit configured or connected via a communication interface (configured to execute the method by software and / or firmware execution); (ii) firmware and / or hardware logic circuits integrated into electronic equipment, which utilize hardware acceleration or special processing capabilities not typically provided by general-purpose processing units; or (iii) an operation involving one or more chiplets within one or more integrated circuit packages. Some implementations of this technique involve a device acting as a switch, which consists of switching circuits and is configured to execute the method.

[0108] In computing environments where a host communicating according to CXL.mem needs to access memory resources within the UALink domain, such as memory connected to an accelerator or memory accessible via the UALink network, the RPU may bridge the two protocol domains by translating between CXL.mem and UPLI. The RPU may include a CXL port for communicating with the host and a UALink port for communicating with accelerators and UALink switches. The RPU may translate CXL.mem M2S requests into UPLI requests and UPLI responses into CXL.mem S2M responses. The RPU may include a cache for storing acquired data and may operate as a CXL type 2 or CXL type 3 device. The RPU may be a separate component, an IP block embedded within an accelerator, or a chiplet within an IC package.

[0109] In various implementations, the system may consist of a Resource Provisioning Unit (RPU) consisting of a Compute Express Link (CXL) port configured to communicate with a first entity according to CXL.mem, and an Ultra Accelerator Link (UALink) port configured to communicate with a second entity according to an UALink-based protocol; where the RPU is configured as follows: it receives a CXL.mem Master-to-Subordinate (M2S) request from the first entity via the CXL port; it translates the CXL.mem M2S request into a UPLI request; and it sends the UPLI request to the second entity via the UALink port. The RPU may act as a protocol translation bridge, allowing entities in the CXL domain to access resources in the UALink domain. The CXL port may support CXL.mem M2S and S2M channels, while the UALink port may support UPLI requests, read responses / data, write responses, and originator data channels. The RPU can perform translations of opcodes, addresses, tags, and additional fields between CXL.mem and UPLI. The first entity includes a CXL host such as a CPU, while the second entity includes accelerators, UALink switches, or other UALink-enabled devices. The RPU may be coupled directly to the first entity or via the CXL fabric, and may be coupled directly to the second entity or via a UALink switch. The system may be implemented within an IC package, between IC packages, or as a combination of individual and integrated components.

[0110] In some implementations of the system, the RPU is further configured as follows: It receives a UPLI read response (RdRsp) from a second entity via the UALink port. It converts the RdRsp of the UPLI into a data response (S2M DRS) from a sublogue in CXL.mem to the master. It sends the CXL.mem S2M DRS to the first entity via the CXL port. The RPU can also convert the RdRsp field of the UPLI into a CXL.mem S2M DRS field, map the RdRspTag to the corresponding CXL.mem tag, and format the RdRspData as CXL.mem data. The RPU can accumulate data from multiple UPLI response beats before constructing the S2M DRS.

[0111] In some system implementations, the RPU further configures a cache. The RPU also stores data from the UPLI RdRsp in its cache and is configured to send a No Data Response (S2M NDR) from CXL.mem to the master to the initial entity. By holding the retrieved data in the cache, the RPU participates in consistency management with the initial entity and can respond to subsequent accesses to the same cache line without issuing additional UPLI requests. A cached RPU may also operate as a CXL type 2 device.

[0112] In some implementations of the system, the first entity consists of the CXL host and the second entity consists of the accelerator, and the RPU is configured to operate as at least one of either a CXL type 2 or CXL type 3 device. The RPU can operate as a CXL type 2 device utilizing the HDM-D memory region with device-managed coherency, or as a CXL type 3 device utilizing the HDM-H memory region with host-managed coherency. In some examples, the RPU may expose both HDM-D and HDM-H regions simultaneously, operating as a CXL type 2 device in some memory regions and as a CXL type 3 device in others.

[0113] In some implementations, a UALink switch and a UALink port may also be combined. In this case, the RPU is configured to send UPLI requests to a second entity via the UALink switch. The UALink switch can route the UPLI request to the second entity based on the Destination Accelerator ID included in the UPLI request. To entities within the UALink domain, the RPU may be perceived as a UALink endpoint identified by its accelerator ID.

[0114] In some implementations of the system, the RPU is further configured as follows: it receives a CXL.mem Master-to-Subordinate Request (M2S RwD) with data containing MemWr* from the first entity via the CXL port; it translates the CXL.mem M2S RwD into a UPLI request containing write commands; and then sends the UPLI request and write data to the second entity via the UALink port. The RPU can translate the write opcode of CXL.mem into a write command in UPLI and transfer the write data from the CXL.mem domain to the UPLI originator data channel. If MemWr* refers to a MemWrPtl in a partial write, the RPU may translate the CXL.mem byte into a UPLI OrigDataByteEn field.

[0115] In computing environments where a host, such as a CPU, needs to access memory resources residing in the UALink domain (e.g., HBM or HBF connected to accelerators within a UALink pod), the host can communicate with the RPU via CXL.mem, translate requests to UPLI, and deliver them to the accelerators via the UALink switch. The UALink pod may also contain accelerators connected to the UALink switch, forming a high-bandwidth interconnect fabric for inter-accelerator communication. The RPU may be embedded within one of the accelerators or be a separate component connected to the UALink switch. Some configurations include a CXL fabric connecting the host to the RPU and CXL memory devices such as GFDs, memory expanders, and memory pools, allowing the host to access both CXL domain resources and UALink domain resources. Multipath topologies allow the host to access different accelerators and memory resources through different translation paths, supporting workloads that incrementally migrate data from CXL memory to accelerator memory, such as AI inference.

[0116] In various implementations, the following systems exist: a host; an UALink pod combining an UALink switch and accelerators, where accelerators communicate with each other according to an UALink-based protocol; and a Resource Provisioning Unit (RPU) connected to the host via CXL.mem and working in conjunction with the UALink switch. The RPU receives subrosciate (M2S) requests from the host's CXL.mem master, translates the CXL.mem M2S requests into UPLI requests, and is configured to send UPLI requests to its accelerators via the UALink switch. This system may allow the host to access memory resources in the UALink domain (e.g., HBM or HBF linked to the accelerators), DRAM within the accelerators, or memory accessible via the UALink network. The RPU can perform translations of opcodes, addresses, tags, and additional fields between CXL.mem and UPLI. The UALink switch can route UPLI requests to target accelerators based on the destination accelerator ID. The host can utilize CXL.mem M2S requests containing memory read / write opcodes, which the RPU can translate into corresponding UPLI read / write commands. The system may support workloads where the host steps data between CXL memory resources and accelerator memory, such as populating the accelerator HBM or HBF with KV cache entries for AI inference workloads. The RPU may be implemented as a standalone component, as an IP block embedded within the accelerator, or as a chiplet within an IC package.

[0117] In some system implementations, the RPU is embedded within the accelerator's first accelerator, which is connected to the host via CXL.mem. UPLI requests are sent from the first accelerator to the second accelerator via the UALink switch. The embedded RPU may share the first accelerator's silicon die or IC package and utilize the first accelerator's CXL port for communication with the host. The first accelerator may function as an accelerator within the UALink pod and act as a translation bridge for the host to reach other accelerators.

[0118] In some system implementations, the RPU is separate from the accelerator and connected to the UALink switch via a UALink port. Discrete RPUs can also function as independent protocol conversion components that do not perform accelerator computing functions. This configuration allows the RPU to be deployed independently of the accelerator design and potentially support different generations of accelerators and different UALink pod configurations.

[0119] Some implementations also include a CXL fabric that couples the host and the RPU. Furthermore, it includes CXL memory devices connected to the host via the CXL fabric, with the host configured to access the CXL memory devices through CXL.mem. The CXL fabric may include one or more CXL switches providing connectivity between the host, RPU, and CXL memory devices. The host can access the CXL memory devices via CXL.mem without protocol translation, while the RPU can access accelerator memory within the UALink pod via protocol translation.

[0120] In some system implementations, a CXL memory device consists of at least one: a Global Fabric Attached Memory Device (GFD), a CXL memory expander, or a CXL memory pool. The GFD provides a large memory resource accessible via CXL.mem and may be shared among hosts and other CXL entities. The host can stage data between CXL memory devices and accelerator memory via the RPU, enabling tasks such as migrating KV cache entries for AI inference workloads.

[0121] In some implementations of the system, an RPU is further configured to receive UPLI read responses (RdRsp) from the accelerator via the UALink switch, convert the UPLI RdRsp to a CXL.mem Subordinate-to-Master data response (S2M DRS), and send the CXL.mem S2M DRS to the host. The host consists of a central processing unit (CPU), and the accelerator consists of a graphics processing unit (GPU) or a tensor processing unit (TPU). The CPU can use the CXL.mem responses to input data structures, move data between memory hierarchies, and perform orchestration tasks for AI and HPC workloads. GPUs and TPUs can also function as computation accelerators within the UALink pod.

[0122] In some implementations, the system is further connected to the host via CXL.mem and includes a second RPU connected to the UALink switch. In this case, the RPU is configured to translate CXL.mem M2S requests from the host into UPLI requests targeting the first accelerator of the accelerator, and the second RPU is configured to translate CXL.mem M2S requests from the host into UPLI requests targeting the second accelerator of the accelerator. The RPUs allow the host to access different accelerators through dedicated translation paths and may support different address ranges, different coherency policies, or different quality of service (QoS) configurations for each path. The first and second RPUs may be independent components or may include both embedded and discrete RPUs.

[0123] In some system implementations, the UALink switch is configured with a route table, which routes UPLI requests from the RPU to the accelerator based on the destination accelerator ID included in the UPLI request. The accelerator also communicates with the UALink switch via the UPLI request channel and the UPLI response channel. The UALink switch routes UPLI traffic based on the ReqDstPhysAccID field of each UPLI request, using the route table to map the Destination Accelerator ID to the exit port connected to the target accelerator. The route table may be programmed by the pod controller or other management body. The RPU may input the ReqDstPhysAccID based on the address of the CXL.mem M2S request or a pre-configured mapping.

[0124] Figure 9A shows an example block diagram where a computer (Entity.3) connects the first entity (Entity.1) and the second entity (Entity.2). The first entity communicates with the computer via CXL.mem, and the computer communicates with the second entity (Interface.2) via an UALink-based protocol. The first entity includes a CXL host, switch, or consumer of memory resources. The second entity includes an accelerator, switch, or provider of memory resources. The computer may include an xPU, bridge, switch, or RPU.

[0125] Figure 9B shows an example of how TFD converts a CXL.mem request from entity.1 to a UPLI request from entity.2, and then converts the UPLI read response from entity.2 to a CXL.mem S2M DRS sent to entity.1. The CXL.mem M2S request contains the MemOpcode, Tag, and Address fields, while the UPLI request contains the ReqCmd, ReqSrcPhysAccID, ReqDstPhysAccID, ReqAddr, and ReqTag fields. The UPLI read response contains the RdRspDstPhysAccID, RdRspTag, and RdRspData fields, while the CXL.mem S2M DRS holds the Opcode, Tag, and Data fields.

[0126] Figure 10A shows an example of a TFD demonstrating an RPU including a cache (cache with RPU) that exposes a CXL Type-2 device using the HDM-D memory region. Entity.1 sends a CXL.mem M2S request combining SnpType(SnpData), MetaField(MS0), and MetaValue(S) to request a shared copy of the cache line. The RPU translates the request into a UPLI request and receives a UPLI read response from Entity.2. The RPU stores the fetched data in the cache and sends a CXL.mem S2M NDR including Cmp-S to Entity.1, which indicates the shared cache line state. It then sends a CXL.mem S2M DRS containing MemData and the data.

[0127] Figure 10B shows an example of a TFD illustrating an RPU without a cache, exposing a CXL Type-3 device utilizing the HDM-H memory region. Entity.1 sends a CXL.mem M2S request containing SnpType(No-Op) and MetaField(No-Ops). The RPU translates the request into a UPLI request and receives a UPLI read response from Entity.2. The RPU translates the UPLI read response into a CXL.mem S2M DRS containing MemData and data, and sends the S2M DRS to Entity.1 without sending an S2M NDR.

[0128] Figure 11 shows an example system consisting of a host CPU, CXL fabric, UALink switch, accelerator, and RPU. Path 1 (1a to 1b) shows the host including an embedded RPU, the host communicating with Accelerator.n via CXL.mem, and Accelerator.n communicating with Accelerator.1 via the UALink switch. Path 2 (2a to 2b) shows the host accessing Accelerator.k via a discrete RPU (RPU.1) connected to the UALink switch. The host communicates with RPU.1 via CXL.mem, and RPU.1 communicates with Accelerator.k via UALink. Path 3 (3a to 3b) shows the host accessing the GFD through the CXL fabric via CXL.mem.

[0129] In computing environments where a host, such as a CPU, accesses memory resources on devices such as accelerators, devices may expose memory regions to the host via CXL. Different memory regions may have different coherency requirements and may be backed by different types of memory. For example, the first memory region may be backed by local memory coupled to a device, such as HBM or high-bandwidth flash (HBF), and device coherency may benefit if the device participates in host-cache consistency. The second memory region may be backed by memory accessible via the UALink network, such as memory residing on a remote accelerator, and may not require device coherency participation. The CXL specification defines flows for different HDM types and device types, each corresponding to a different coherency model. Devices can also expose parallel HDM regions using flows for different device types. The RPU and translation logic within the device maintain appropriate coherency semantics for each memory region while translating CXL protocol messages received from the host between UPLI messages for memory access within the UALink domain.

[0130] Various implementations employ the following techniques: a device connected to a host via a Compute Express Link (CXL) link exposes a first memory region via a first CXL device type flow and a second memory region via a second CXL device type flow, where the first CXL device type flow is distinct from the second. Here, the first memory region is associated with the first memory. Here, the second memory region is associated with a second memory accessible via an Ultra Accelerator Link (UALink) based protocol. Depending on the device, at least one of the first or second memory regions is translated between CXL and UALink protocol-level interface (UPLI) based protocols. This method can be implemented in hardware, firmware, software, or a combination thereof and may be performed by various devices such as accelerators, RPUs, semiconductor devices, and chiplets within IC packages. The first and second CXL device type flows correspond to any combination of CXL type 2 and type 3 device flows, and may also include CXL type 1 device flows in some cases. Devices expose additional memory regions beyond the first and second memory regions, each using different or the same CXL device type flows. Translations between CXL and UPLI-based protocols may include translations of opcodes, addresses, tags, and additional fields, and may also include translations between different address spaces, such as the host physical address (HPA) space and the network physical address (NPA) space. The first memory includes memory bound to devices such as HBM, HBF, DRAM, and GDDR, while the second memory includes memory accessible through UALink switches, UALink networks, or remote accelerators within a UALink domain. Elements may communicate through one or more intermediary components, such as switches, retimers, or other appropriate entities that facilitate information transfer.

[0131] In some implementations of this method, the first CXL device type flow consists of a CXL type 2 device flow, where the first memory region is a host-managed device memory (HDM-D) region with device coherency, and the second CXL device type flow is a CXL type 3 device flow, where the second memory region is a host-managed device memory (HDM-H) region with host-only coherency. The device then participates in cache coherency with the host in the first memory region but not in cache coherency with the host in the second memory region. The CXL Type-2 device flow allows the device to utilize both the CXL.mem and CXL.cache protocols in the HDM-D region, maintain a cached copy of the data, and participate in coherency negotiations with the host. The CXL Type-3 device flow may utilize CXL.mem in the HDM-H region where the host manages coherency without device cache participation.

[0132] In some implementations, the host sends a CXL.mem Master-to-Subordinate (M2S) request consisting of MemRd* and the address targeting the first memory region, which further includes SnpType, MetaField, and MetaValue fields. The CXL.mem M2S request is converted into a UPLI request; upon receiving the UPLI response containing the data, the device sends back to the host a CXL.mem Subordinate-to-Master Data Response (S2M NDR) consisting of Cmp-S or Cmp-E indicating the cache state of the cache line on the address, and a CXL.mem Subordinate-to-Master Data Response (S2M DRS) containing MemData and the data. The SnpType, MetaField, and MetaValue fields in the CXL.mem M2S request may indicate the host's cache line state intention, such as a shared copy (SnpData) or exclusive copy (SnpInv) request. The device can use these fields to determine the appropriate coherence response. The Device Coherency Engine (DCOH) may select Cmp-S if the device holds a cached copy of the data, and Cmp-E if the device abandons the cached copy. Before responding, the device can convert the CXL.mem M2S request into a UPLI request to retrieve data from the UALink domain.

[0133] In some implementations of this technique, the device is configured with a cache, and the device stores the data of the UPLI response in the cache and sends a CXL.mem S2M NDR consisting of Cmp-S to indicate that it holds a cached copy of the cache line at the address. By caching the retrieved data and responding with Cmp-S, the device can provide subsequent access to the same cache line from its local cache, without requiring additional UPLI transactions. A device with a cache, or a device that controls or utilizes a cache, may include cache memory, a cache controller, and cache allocation and eviction logic.

[0134] In some implementations of this method, a UPLI request consists of the ReqSrcPhysAccID, ReqDstPhysAccID, ReqTag, ReqAddr, and ReqCmd fields, which contain the read command. Furthermore, it includes converting the tags of the CXL.mem M2S request to the ReqTag of the UPLI request. The ReqSrcPhysAccID and ReqDstPhysAccID fields may contain identifiers used by the UALink network to route UPLI requests. Tag conversion involves maintaining a bidirectional mapping between CXL.mem Tag values ​​and UPLI ReqTag values, enabling proper correlation between UPLI responses and corresponding CXL.mem requests.

[0135] In some implementations, the device receives a CXL.mem Master-to-Subordinate (M2S) request from the host, which includes MemRd* and addresses targeting a second memory region. The CXL.mem M2S request is translated into a UPLI request; upon receiving the UPLI response containing the data, the device sends MemData and a CXL.mem Subleg-to-Master Data Response (S2M DRS) containing the data to the host. In the second memory region, the device acts as a pass-through translator, retrieving data from the UALink domain and returning it to the host without maintaining a cached copy or participating in coherence negotiations. The CXL.mem S2M DRS can carry MemData without an S2M NDR indicating Cmp-S or Cmp-E because the device does not track the cache state of this memory region.

[0136] In some implementations of this technique, the UPLI response may include an RdRspDataError field indicating a data error. Furthermore, this may involve translating the RdRspDataError field into a Poison field in the CXL.mem S2M DRS sent to the host. The RdRspDataError field in the UPLI response can function as a per-beat data poison indicator. By translating the error indication across protocol boundaries, the host can detect data corruption occurring in the UALink domain and take appropriate recovery actions.

[0137] In some implementations of this technique, the device communicates with the host via CXL.cache for the initial memory region. The device sends a CXL.cache Device-to-Host (D2H) request to the host containing an opcode selected from RdOwn, RdShared, RdCurr, or RdAny. The CXL.cache D2H request allows the device to initiate a coherency transaction with the host for the data in the initial memory region. RdOwn can obtain exclusive ownership, RdShared can obtain a shared copy, RdCurr can request a non-cacheable current value, and RdAny can accept any coherency state.

[0138] In some implementations of this approach, the first memory consists of at least one high-bandwidth memory (HBM) or high-bandwidth flash (HBF) coupled to the device, and the second memory consists of memory accessible through UALink switches or UALink networks, with the device being an accelerator. The accelerator is a GPU, TPU, or other processing unit with HBM and / or HBF, which may benefit from the device coherence of the local memory. The UALink switch or fabric connects the accelerator to a remote accelerator, and the second memory may reside in the remote accelerator or other memory resources within the UALink domain.

[0139] In some implementations, this technique also includes the ability for the device to translate between a first address associated with the first address space used by the host and a second address associated with the second address space used by the UALink-based protocol. Here, the first address space consists of the host physical address (HPA) space, and the second address space consists of the network physical address (NPA) space or the system physical address (SPA) space. Address translation can be implemented using lookup tables, page tables, base offset calculations, or programmable translation functions. The HPA space may represent a representation of the host's memory, while the NPA and SPA spaces may represent addresses used by the UALink network for routing and accessing memory resources.

[0140] In some implementations of this technique, at least one of the first or second memory regions consists of host-managed device memory with back-invalidation (HDM-DB). The device sends a CXL.mem Subordinate-to-Master Back-Invalidate Snoop (S2M BISnp) to the host, and the host responds with a CXL.mem Master-to-Subordinate Back-Invalidate Response (M2S BIRsp). The HDM-DB region may allow the device to snoop the host cache when it needs to modify or evict cached data. S2M BISnp carries opcodes such as BISnpInv, BISnpData, and BISnpCur, while M2S BIRsp carries opcodes such as BIRspI, BIRspS, and BIRspE, indicating the resulting host cache state. HDM-DB is available in either CXL Type-2 or CXL Type-3 device flows.

[0141] In some implementations, the host receives a master-to-subless request (M2S RwD) for CXL.mem, which may include MemWr* and write data. The CXL.mem M2S RwD is converted into a UPLI request consisting of write commands and write data; a subless-to-master no-data response (S2M NDR) for CXL.mem is sent to the host. The write command in the UPLI request may include a Write command or a WriteFull command. Depending on the order requirements and system configuration, the device may send an S2M NDR before or after the completion of the UPLI write.

[0142] Some implementations of this method consist of instructions in a non-temporary, computer-readable medium that, when executed by a processor, cause the processor to execute the method. Some implementations of this technique involve one or more integrated circuits configured to execute this technique, and these one or more integrated circuits constitute one of the following: (i) a general-purpose processing unit configured or connected via a communication interface (configured to execute the method by software and / or firmware execution); (ii) firmware and / or hardware logic circuits integrated into electronic equipment, which utilize hardware acceleration or special processing capabilities not typically provided by general-purpose processing units; or (iii) the operation of having one or more chiplets within one or more integrated circuit packages.

[0143] In computing systems where a host accesses memory resources on a CXL-coupled device, the device may expose memory regions with different coherency characteristics to the host. The first memory region, related to local memory (such as HBM), is exposed through a flow of a CXL device type that supports device coherency, allowing the host and device to maintain a consistent cached copy of the data. A second memory region, related to memory accessible via the UALink port, may be exposed through a flow of another CXL device type that does not require device coherency participation. This device may incorporate an RPU or translation logic that performs translation between CXL protocol messages and UPLI messages for memory access operations targeting UALink-accessible memory. The UALink switch may connect the device to one or more remote accelerators, and the memory resources of those accelerators may form the second memory region.

[0144] In various implementations, the following system exists: a host; a device connected to the host via a Compute Express Link (CXL) link; and a first memory region connected to the device. This device is configured to expose a first memory region to the host via a first CXL device type flow, and a second memory region via a second CXL device type flow, where the first CXL device type flow is distinct from the second. Here, the first memory region is associated with the first memory. Here, the second memory region is associated with a second memory region accessible from the device's Ultra Accelerator Link (UALink) port. The device is also configured to translate between CXL-based protocols and the UALink protocol-level interface (UPLI), and to handle requests targeting at least either the first or second memory region. This system allows the host to access both local and remote memory resources on the device via the CXL link, and enables distinct coherency semantics for each different memory region. This device may include an RPU, translation logic, or a combination of hardware and firmware that performs the translation between CXL and UPLI. The device can dynamically or statically define the boundary between the first and second memory regions, for example, by utilizing HDM decoder registers or programmable address range registers.

[0145] In some system implementations, the first CXL device type flow consists of a CXL type 2 device flow, with the first memory region being host-managed device memory (with device coherency), and the second CXL device type flow consists of a CXL type 3 device flow, with the second memory region being host-managed device memory (host-only coherency) region (HDM-H). CXL type-2 device flows may be able to negotiate CXL.io, CXL.cache, and CXL.mem in the HDM-D region, while CXL type-3 device flows can cross CXL.io and CXL.mem in the HDM-H region. In some examples, the allocation of HDM type memory regions may be configurable during system initialization or runtime.

[0146] In some system implementations, for CXL.mem requests targeting the first memory region, the device is configured to send a CXL.mem Subordinate-to-Master No Data Response (S2M NDR) consisting of Cmp-S or Cmp-E, indicating the cache state. If the CXL.mem request targets the second memory region, the device is configured to convert the CXL.mem request to a UPLI request and send a CXL.mem Subordinate-to-Master Data Response (S2M DRS) containing MemData. The differentiated response behavior may reflect different coherency models for the first and second memory regions. For the first memory region, the display of Cmp-S or Cmp-E can inform the host of the cache state of the cache line on the device. For the second memory region, the device may convert the request to a UPLI, retrieve data from the UALink domain, and return the data.

[0147] In some implementations of the system, the host communicates with the device via CXL.mem and CXL.cache for the first memory region, while the host communicates with the device without CXL.cache. Using CXL.cache for the first memory region allows the device to initiate coherence transactions, respond to host snoops, and support scenarios where both the device and host can cache data from the first memory region. The absence of CXL.cache for the second memory region may simplify the memory access path to remote memory.

[0148] In some implementations of the system, the device consists of an accelerator containing a Resource Provisioning Unit (RPU), with the first memory being at least one High Bandwidth Memory (HBM) or High Bandwidth Flash (HBF) coupled to the accelerator. Furthermore, a UALink switch connects UALink ports to one or more remote accelerators, and the second memory is accessible through the UALink switch. The RPU may be implemented as an IP block embedded within the accelerator, or as a chiplet within the IC package containing the accelerator. The UALink switch can route UPLI traffic based on the destination accelerator identifier included in the UPLI request. Each of the one or more remote accelerators has its own HBM, HBF, or other memory, which together form the second memory accessible from the device.

[0149] In computing environments where a host, such as a CPU, accesses the memory resources of CXL-connected devices, devices may expose memory regions with different connectivity to the host. The first memory region may be backed by local memory connected to the device, while the second memory region may be backed by memory accessible via the NVLink fabric, such as memory on a GPU or other NVLink-connected devices. NVLink provides high-bandwidth communication between GPUs and accelerators and can support a distributed memory model where devices access memory through other devices. Devices translate CXL protocol messages received from the host into NVLink messages for memory access within the NVLink domain, exposing different CXL device type flows for each different memory region to provide appropriate coherence semantics. NVLink messages may include fields such as source and destination identifiers for routing, memory location addresses, transaction tags for response correlation, a length field for the transfer size, and a data payload.

[0150] Various implementations employ the following technique: a device connected to a host via a Compute Express Link (CXL) link exposes a first memory region via a first CXL device type flow and a second memory region via a second CXL device type flow, where the first CXL device type flow is distinct from the second. Here, the first memory region is associated with the first memory. Here, the second memory region is associated with the second memory accessible via an NVLink-based protocol. The device then translates between the CXL-based and NVLink-based protocols in at least one of the first or second memory regions. This method can be implemented in hardware, firmware, software, or a combination thereof and may be performed by various types of devices, such as accelerators, RPUs, semiconductor devices, active cables, and chiplets within IC packages. The first and second CXL device type flows may correspond to any combination of CXL type 2 and type 3 device flows. The translation between CXL and NVLink may include the translation of opcodes, addresses, transaction identifiers, and additional fields. NVLink messages may have functional fields corresponding to the source identifier, destination identifier, address, transaction tag, transfer length, and data payload. Specific field names may vary depending on the NVLink version and implementation, and translations may accommodate such differences. The first memory may contain memory coupled to devices such as HBMs and HBFs, while the second memory may contain memory accessible from GPUs and other NVLink-connected devices. Devices may reside in active cables, modules connected to CXL ports, or within computing platforms, bridging the gap between the CXL domain and the NVLink domain. Elements may communicate through one or more intermediary components, such as NVLink switches or other appropriate entities facilitating information transfer.

[0151] In some implementations of this technique, the first CXL device type flow consists of a CXL type 2 device flow, where the first memory region is a host-managed device memory (HDM-D) region with device coherency, and the second CXL device type flow consists of a CXL type 3 device flow, where the second memory region is a host-managed device memory (HDM-H) region with host-only coherency. The device then participates in host and cache coherency in the first memory region but not in host and cache coherency in the second memory region. The CXL Type-2 device flow allows the device to maintain a cached copy of data from the first memory region or participate in coherency negotiations with the host via CXL.cache. The CXL Type-3 device flow for the HDM-H region may make pass-through access to NVLink-accessible memory easier without the device coherency overhead.

[0152] In some implementations, this method receives a CXL.mem Master-to-Subordinate (M2S) request from the host, which includes the first address targeting the MemRd* and the second memory region. The CXL.mem M2S request is then translated into an NVLink read request consisting of SourceID, DestinationID, the second address, a tag, and a length; upon receiving an NVLink read response, *Data*; the host receives a CXL.mem subordinate-to-master data response (S2M DRS) containing MemData and the NVLink read response data. The SourceID identifies the device or RPU that initiated the NVLink read request, and the DestinationID can identify the target entity, such as a GPU, within the NVLink fabric. The second address is the NVLink network address, used to route the NVLink read request to its destination, and may undergo additional address translation phases facilitated by one or more Link TLBs within the NVLink domain. The tag may be a transaction identifier managed by the device to associate the NVLink read response with the original CXL.mem M2S request. The length may indicate the requested transfer size. In NVLink read responses, *Data* may represent data contained in one or more response packets. Depending on the NVLink version and implementation, different naming conventions may be used for these function fields. For example, the source identifier is called the requester identifier, source node identifier, or similar designation, and the destination identifier is called the target identifier, destination node identifier, or similar designation.

[0153] In some implementations, this technique also involves receiving a CXL.mem Master-to-Subordinate (M2S) request from the host consisting of a MemRd* and an address targeting the first memory region. Accessing the first memory region and retrieving data; the device sends the host a CXL.mem subordinate-to-master data response (S2M NDR) consisting of Cmp-S or Cmp-E indicating the cache state of the cache line on the address, and a CXL.mem subordinate-to-master data response (S2M DRS) containing MemData and the data. In the first memory region, the device can access local memory such as HBM or HBF without protocol translation to NVLink. The device may respond with Cmp-S or Cmp-E based on the device's cache policy and the requested coherency state of the host, as indicated by the SnpType and MetaValue fields in the M2S request.

[0154] In some implementations of this technique, the device communicates with the host via CXL.cache for the initial memory region. The device sends a CXL.cache Device-to-Host (D2H) request to the host, containing an opcode selected from RdOwn, RdShared, RdCurr, or RdAny. The CXL.cache D2H request allows the device to initiate a coherence transaction with the host, for example, when the device needs to read or modify data cached by the host.

[0155] In some implementations, this technique also involves the device translating a first address associated with the physical address space (HPA) used by the host to a second address associated with the NVLink network address space used by NVLink-based protocols. Address translation can be implemented using lookup tables, page tables, base offset calculations, or programmable translation functions. The NVLink network address can be used to route NVLink transactions to specific GPUs or memory resources within the NVLink fabric.

[0156] Some implementations of this method consist of instructions in a non-temporary, computer-readable medium that, when executed by the processor, cause the processor to execute the method.

[0157] In computing systems where a host accesses memory resources on a device via CXL, the device may expose memory regions with different connectivity and coherency models. The first memory region is backed by local memory coupled to the device and may be exposed in flows of CXL device types that support device coherency. The second memory region is backed by memory accessible via the NVLink port, such as memory on a GPU or other NVLink-connected device, and may be exposed in flows of a different CXL device type. This device may have an RPU or translation logic built in to perform NVLink-accessible memory operations using translation between CXL protocol messages and NVLink messages. An NVLink switch, such as an NVSwitch, connects the device to one or more GPUs, and the memory resources of those GPUs form the second memory region.

[0158] In various implementations, the following system exists: a host; a device connected to the host via a Compute Express Link (CXL) link; and a first memory connected to the device, which is configured to expose a first memory region to the host via a first CXL device type flow and a second memory region via a second CXL device type flow, where the first CXL device type flow is distinct from the second CXL device type flow. Here, the first memory region is associated with the first memory. Here, the second memory region is associated with the second memory accessible through the device's NVLink port. The device is configured to translate requests between CXL-based and NVLink-based protocols for at least one of the first or second memory regions. This system allows the host to access local and NVLink domain memory resources on the device via the CXL link, and can have differentiated coherence semantics for each different memory region. The device may include an RPU, translation logic, or a combination of hardware and firmware that performs translation between CXL and NVLink-based protocols. The device may be an accelerator, RPU, bridge device, or a component in an active cable located between the CXL domain and the NVLink domain. The device can dynamically or statically define the boundary between the first and second memory regions, for example, by utilizing HDM decoder registers or programmable address range registers. This system could be deployed in data center environments where CXL-enabled CPUs work in conjunction with NVLink GPUs for AI model inference and training.

[0159] In some implementations of the system, the first CXL device type flow consists of a CXL type 2 device flow, with the first memory region being host-managed device memory (with device coherency), and the second CXL device type flow consists of a CXL type 3 device flow, with the second memory region being host-managed device memory (host-only coherency) region (HDM-H). CXL type-2 device flows may be able to negotiate CXL.io, CXL.cache, and CXL.mem in the HDM-D region, while CXL type-3 device flows can cross CXL.io and CXL.mem in the HDM-H region.

[0160] In some system implementations, for CXL.mem requests targeting the first memory region, the device is configured to send a CXL.mem Subordinate-to-Master No Data Response (S2M NDR) consisting of Cmp-S or Cmp-E, indicating the cache state. If the CXL.mem request targets the second memory region, the device is configured to translate the CXL.mem request into an NVLink read request and send a CXL.mem Subordinate-to-Master Data Response (S2M DRS) containing MemData. The differentiated response behavior may reflect different coherency models for the first and second memory regions. In the first memory region, the Cmp-S or Cmp-E instruction can inform the host of the cache state maintained by the device. In the second memory region, the device translates the request into an NVLink read request, receiving data from the NVLink domain and sending it back to the host.

[0161] In some implementations of the system, the device consists of an accelerator or resource provisioning unit (RPU), and the first memory is at least one high-bandwidth memory (HBM) or high-bandwidth flash (HBF) coupled to the device. Furthermore, an NVLink switch connects the device's NVLink ports to one or more GPUs, and the second memory is accessible through the NVLink switch. The NVLink switch is an NVSwitch or similar switch device that provides high-bandwidth routing between the device and GPUs within the NVLink fabric. One or more GPUs each have HBM, HBF, or other memory, and these together form a second memory accessible from the device via the NVLink port.

[0162] Figure 12A shows an example of a system consisting of an active cable including an RPU. The cable consists of a first pluggable module (Module 1) and a second pluggable module (Module 2) coupled by a physical medium. Module 1 contains the RPU and is connected to the CXL port of the first entity (Entity 1) via the first electrical connector (Electrical Connector 1). Module 2 is connected to the NVLink port of the second entity (Entity 2) via the second electrical connector (Electrical Connector 2). Entity 1 may be a CXL host, CPU, GPU, CXL Switch, MxPU, or consumer. Entity 2 may be a GPU, CPU, accelerator, NVLink switch (e.g., NVSwitch), or provider. The RPU may be located in various places depending on the requirements. As an example, the RPU is located in Module 1, close to the CXL port of Entity 1. This is because CXL, operating on PCIe electrical circuitry, is designed as a short-range interface for connecting to CPUs within the computing platform. Some versions of NVLink incorporate electrical signaling characteristics compatible with Ethernet and / or InfiniBand connectivity and are designed for rack-level deployments and longer-distance interconnects. Placing the RPU closer to the CXL port can potentially improve signal integrity. Furthermore, because NVLink typically uses higher signal speeds than CXL, it may require fewer lanes for the same bandwidth, thus reducing the amount of copper or fiber optics in the physical medium.

[0163] Figure 12B shows an example of a TFD (Transform Flow Diagram) that translates an RPU between a CXL.mem request and an NVLink request. The TFD displays three entities: Entity.1 / Consumer on the left, RPU in the center, and Entity.2 / Provider on the right. Entity.1 can send a CXL.mem M2S Req to the RPU, consisting of a MemOpcode (MemRd), Addr (AS.1.1), and Tag (p.1.1). The Address (AS.1.1) may be the host's HPA, or, for example, a CXL-enabled CPU connected to the RPU. The RPU can translate the CXL.mem M2S request into an NVLink request read consisting of a SourceID (a.1), DestinationID (b.1), Address (AS.2.1), Tag (c.1), and Length (d.1). The Address (AS.2.1) is the NVLink network address used to route the NVLink request to a destination on the NVLink fabric. In response, Entity.2 can send an NVLink response to the RPU consisting of SourceID(b.1), DestinationID(a.1), Tag(c.1), and *Data*. The RPU can translate the NVLink response into a CXL.mem S2M DRS consisting of Opcode(MemData), Tag(p.1.1), and Data(*Data*), and send the CXL.mem S2M DRS to Entity.1. The RPU may map the tag from the NVLink response to the original CXL.mem tag(p.1.1), enabling proper transaction completion at Entity.1. In some cases, the NVLink network address may undergo an additional address translation phase, which may be facilitated by the presence of one or more link TLBs on the transaction path. For example, on a GPU, a Link TLB may translate the NVLink network address to a GPU physical address, from which it may reference memory resources integrated with or adjacent to the destination GPU.The RPU may perform another address translation to translate the HPA used in CXL.mem to an NVLink network address before sending an NVLink request. Furthermore, NVLink provides a distributed memory model in which GPUs can access memory through other GPUs. This example provides a general-purpose CXL.mem bridge / gateway for an x86 GP-CPU to access memory on the NVLink fabric, which participates in AI model inference and training together with an NVLink GPU.

[0164] In AI inference systems, accelerators such as GPUs and TPUs may generate and consume large amounts of inference context data. This includes key-value (KV) cache data, model weight parameters, activation tensors, and embedding vectors. If the amount of inference context data exceeds the capacity of the accelerator's local memory, such as HBM, the data may be staged to external memory resources that offer greater capacity at a lower cost, such as CXL memory devices, CXL memory pools, or GFDs. In environments where accelerators are connected via UALink switches and CXL memory devices are accessed via CXL.mem, the RPU performs conversions between UPLI and CXL.mem, allowing accelerators to migrate inference context data between local memory and CXL memory devices across the boundaries of the UALink and CXL protocol domains. Accelerators can initiate migrations by sending UPLI requests to the RPU through the UALink switch, and the RPU translates these requests into CXL.mem M2S requests targeting the CXL memory device. The migration can be bidirectional, with the accelerator writing inference context data to the CXL memory device when evicting data from local memory, and reading inference context data from the CXL memory device if necessary for active computation. The RPU can perform address translation between the address spaces used by the UALink domain and the CXL domain (between NPA or SPA addresses and HPA addresses), and may also perform tag conversion and opcode conversion between UPLI and CXL.mem message formats.

[0165] In various implementations, there are techniques for migrating inference context data across protocol domain boundaries. The following describes an accelerator connecting to an Ultra Accelerator Link (UALink) switch, sending UPLI protocol-level interface (UPLI) requests to a Resource Provisioning Unit (RPU) via the UALink switch, and sending the inference context data when it is stored in the accelerator's local memory. The UPLI request consists of the first physical address, and the RPU translates the UPLI request into a Compute Express Link (CXL) CXL.mem Master-to-Subordinate (M2S) request containing the second physical address; the RPU also sends the CXL.mem M2S request to the CXL memory device; the inference context data is then migrated between the UALink protocol domain and the CXL protocol domain between the accelerator's local memory and the CXL memory device. This technique may be used in AI inference systems where accelerator working memory, such as HBM, cannot simultaneously hold all inference context data. The RPU can perform translations between UPLI and CXL.mem, including conversions of opcodes, commands, addresses, tags, and additional fields. The migration can be performed by the accelerator without host intervention, for example, by determining that certain inference context data is no longer needed and offloading it to a lower-cost memory tier. Alternatively, it may be coordinated by the host or scheduler, instructing the accelerator to purge or retrieve specific data. A UPLI request may include a write command when data is purged from local memory to a CXL memory device that carries the inference context data over the UPLI originator data channel. A UPLI request may also include a read command when data is fetched from the CXL memory device to local memory, in which case the data is returned through the CXL.mem S2M data response path and translated into a UPLI read response.This method can be implemented in hardware, firmware, software, or a combination thereof, and may be executed by various devices such as processors, switches, bridges, RPUs, and semiconductor devices.

[0166] In some implementations of this method, the inference context data consists of key-value (KV) cache data generated during attention computations in a transformer-based inference model, where the KV cache data consists of key and value tensors associated with one or more attention layers in the transformer-based inference model. The KV cache data may increase proportionally to the sequence length and the number of attention layers. As the context length increases, the KV cache consumes a significant portion of the accelerator HBM, incentivizing the migration of recently inaccessible KV cache entries to CXL memory.

[0167] Some implementations of this method utilize at least one transformer-based inference model: (i) Grouped Query Attention (GQA), where multiple query heads share a small number of key-value heads, or (ii) Multi-Latent Attention (MLA), which compresses key and value tensors into a low-rank latent representation; the KV cache data corresponds to a reduced number of key-value heads or a low-rank latent representation, respectively. GQA can reduce KV cache size by sharing key-value heads among multiple query heads, as used in models like Llama. MLA can further compress the KV cache by projecting key and value tensors into a low-dimensional latent space, as used in models like DeepSeek. The reduction in KV cache size per token can impact staging granularity and transfer efficiency.

[0168] In some implementations of this method, the inference context data consists of at least one element: either the model's weight parameters, activation tensors generated during the inference computation, or embedding vectors associated with the input columns. Model weight parameters may be set incrementally when different models or model components are loaded on demand, such as in scenarios involving multiple tenant servers or model switching. Activation tensors may be checkpointed into CXL memory during long inference sequences. Embedding vectors, such as token embeddings or positional embeddings, may be pre-staged from CXL memory before inference begins.

[0169] In some implementations of this method, the accelerator runs a mixed expert inference model consisting of a gating network and expert subnetworks, with the inference context data containing weight parameters for at least one expert subnetwork. It also sends UPLI requests based on the gating network's routing decisions to enable or disable at least one expert subnetwork. In the MoE model, only a portion of the expert subnetworks may be active for any given input token. Inactive expert weights are offloaded to CXL memory, freeing up accelerator HBM capacity, and activated expert weights are retrieved from CXL memory when the gate network routes tokens to experts. This dynamic staging may enable servicing MoE models beyond the available HBM capacity.

[0170] In some implementations, the accelerator also transmits a second UPLI request (a read command and a third physical address) to the RPU via the UALink switch. The RPU converts the second UPLI request into a second CXL.mem M2S request containing a fourth physical address. The second CXL.mem M2S request is transmitted from the RPU to the CXL memory device, and a data response from the CXL.mem subordinate to the master (S2M DRS) is received from the RPU, which consists of secondary inference context data from the CXL memory device. The RPU converts the CXL.mem S2M DRS into a UPLI read response (RdRsp), which consists of the second inference context data. Further, the RPU transmits the UPLI RdRsp to the accelerator via the UALink switch, where the second inference context data is stored in the local memory of the accelerator. The fetch direction is utilized when inference context data previously offloaded to the CXL memory device becomes required for active computation again. The RPU may also convert the CXL.mem S2M DRS, restore the tag to the original UPLI ReqTag, and format the data as UPLI RdRspData. In some examples, such as when the CXL.mem cache line size differs from the UPLI transfer size, the RPU may accumulate data from CXL.mem S2M DRS messages before transmitting the UPLI RdRsp.

[0171] In some implementations of this technique, the CXL memory device refers to a global fabric-attached memory device (GFD), the local memory refers to at least one high-bandwidth memory (HBM) or high-bandwidth flash (HBF), the first physical address refers to a network physical address (NPA) or system physical address (SPA), and the second physical address refers to a host physical address (HPA). The translation includes converting the first physical address to the second physical address. The GFD may be a large-capacity memory accessible by an accelerator via an RPU, and may also be directly accessible by the host via CXL.mem access. Address translation between NPA or SPA and HPA is performed using a lookup table, base-offset calculation, or a programmable translation function in the RPU.

[0172] In some implementations of this method, the active cable consists of first and second pluggable modules coupled by a physical medium, wherein the active cable further constitutes a hardware circuit, is integrated into the active cable, and is configured to perform this method. In some implementations of this technique, an apparatus operating as a switch is disposed, and the apparatus is constituted by a switching circuit and configured to perform the method.

[0173] In AI inference systems, a host, such as a CPU, may orchestrate the staging of inference context data between CXL memory devices residing in the UALink domain and accelerators. The host reads inference context data from CXL memory devices via CXL.mem and sends a CXL.mem M2S request to the RPU to write it to the accelerator. The RPU translates that request into a UPLI request and forwards it to the accelerator via the UALink port. This host-driven staging allows the host to manage a hierarchical memory hierarchy, determine which inference context data to pre-stage to the accelerator based on scheduling policies, inference request queues, and predicted workload patterns, and coordinate data movement between the CXL and UALink protocol domains. The host can also read inference context data from the accelerator via the RPU and write it to the CXL memory device for long-term retention. This bidirectional host orchestrated staging can support a variety of inference architectures and model types, including transformer models with large KV caches, expert hybrid models with dynamic expert activation, distributed prefill and decode architectures, inferential decoding, hybrid attention and state-space models, multimodal models, and search-augmented-generate pipelines. In any case, the CXL memory device acts as an intermediate stage area that bridges the capacity gap between accelerator working memory and the amount of inference context data associated with the workload.

[0174] In various implementations, there are ways to stage inference context data across protocol domain boundaries. This involves the host reading inference context data from a Compute Express Link (CXL) memory device via CXL.mem; the host sending a CXL.mem Master-to-Subordinate (M2S) request to a Resource Provisioning Unit (RPU), which then sends a CXL.mem M2S request containing the first physical address associated with the inference context data; the RPU translating the CXL.mem M2S request into an Ultra Accelerator Link (UALink) protocol-level interface (UPLI) request containing the second physical address; and the RPU then sending the UPLI request to the accelerator via the UALink port; the inference context data is processed at a stage that spans the CXL protocol domain and the UALink protocol domain, from the CXL memory device to the accelerator's local memory. Host-driven staging allows the host to manage a hierarchical memory hierarchy consisting of accelerator local memory as the working memory layer, host memory as the middle layer, and the CXL memory device as the capacity layer. The host can determine which inference context data to stage based on scheduling policies, inference request queues, and forecasts of future workload requirements. The RPU may be a separate component, an IP block embedded in an accelerator, or a chiplet within an IC package. CXL.mem M2S requests may contain write commands carrying inference context data, and translated UPLI requests may convey data to the accelerator over the UPLI originator data channel. The RPU can perform address translation between HPA addresses used by the host and NPA or SPA addresses used in the UALink domain, as well as tag and opcode conversion between CXL.mem and UPLI message formats.The host can access CXL memory devices via CXL.mem without conversion, and can also access accelerators via the RPU, which is converted between CXL.mem and UPLI. This method can be implemented in hardware, firmware, software, or a combination of these.

[0175] In some implementations, this technique further involves the host reading second inference context data from the accelerator. This read involves the host sending a second CXL.mem M2S request to the RPU, which translates the second CXL.mem M2S request into a second UPLI request (including the read command) and sends it to the accelerator via the UALink port. The RPU receives a UPLI read response (RdRsp) from the accelerator consisting of the second inference context data, and the RPU returns the second inference context data to the host. The host writes the second inference context data to the CXL memory device via CXL.mem. This direction allows the host to evicte the inference context data from the accelerator's local memory to the CXL memory device when the data is no longer actively needed or when local memory capacity is exceeded. The host can coordinate both staging and evicting to maintain a working set of inference context data in the accelerator's local memory that matches the current workload.

[0176] In some implementations of the method, the accelerator runs a mixed expert (MoE) inference model consisting of a gating network and an excapical subnetwork, and the second inference context data consists of the weight parameters of the inactive expert subnetwork, i.e., the parameters of the inactive expert subnetwork identified based on the routing decisions of the gating network. By removing the inactive expert weights into CXL memory, the accelerator HBM capacity of the active experts is freed up, enabling support for MoE models where the excapical weight parameters exceed the HBM capacity.

[0177] In some implementations of this technique, the second inference context data consists of key-value (KV) cache entries that have been evicted from the accelerator's local memory based on one of the following: access frequency, recent access, or KV cache entries exceeding the local memory capacity. Long-context inference models may generate KV cache entries that exceed the accelerator's HBM capacity. Eviction policies based on access frequency or recent access may allow the most relevant KV cache entries to be retained in HBM, while less frequently accessed entries can be offloaded to CXL memory and retrieved later.

[0178] In some implementations of this method, inference context data consists of key-value (KV) cache data associated with the trans-based inference model, and the host stages the KV cache data from the CXL memory device to the accelerator's local memory based on scheduled and predictive inference requests. The host maintains a scheduling queue for inference requests and can pre-stage KV cache data associated with future requests to reduce delays when requests are dispatched to the accelerator. Predictions of future requests may be based on session affinity, user activity patterns, and model serving policies.

[0179] In some implementations of this method, the accelerator consists of a decode accelerator, and the inference context data consists of key-value (KV) cache data generated during the prefill phase of the inference operation by the prefill accelerator. The KV cache data is incrementally migrated from the CXL memory device to the local memory of the decode accelerator and used in the decode phase of the inference operation. The CXL memory device acts as intermediate storage between the prefill accelerator and the decode accelerator. In the decomposition inference architecture, the prefill and decode phases are performed by different accelerators to optimize resource utilization. The prefill accelerator writes the generated KV cache data to the CXL memory device, and the host then stages the KV cache data from the CXL memory device to the decode accelerator. Thus, the CXL memory device can act as a shared staging area that separates the prefill and decode phases across protocol domain boundaries.

[0180] Some implementations of this method perform inferential decryption, which consists of a draft model in which the accelerator generates candidate token sequences and a validation model in which it accepts or rejects those candidate token sequences. The inference context data includes either the weight parameters of the draft model, the KV cache data of the draft model, or the validation context data related to inferential decryption. Speculative decryption may use a smaller draft model to generate candidate tokens, which are then accepted or rejected by a larger validation model. The weight parameters of the draft model and the draft model KV cache data may be incrementally migrated from CXL memory to the accelerator HBM when inferential decryption is initiated in an inference session.

[0181] Some implementations of this method involve a hybrid inference model that includes an inference context data set, a state data set associated with the hybrid inference model, an attention layer that generates key-value (KV) cache data, and a state-space model layer that maintains the return state vector. Staging consists of staging at least one KV cache data or recurrent state vector from a CXL memory device into the accelerator's local memory. In hybrid models that combine an attention layer with a Mamba-style state-space model (SSM) layer, both the KV cache data in the attention layer and the recurrent state vector in the SSM layer may be maintained. Staging may contain both data types, each with different sizes, access patterns, and staging priorities.

[0182] In some implementations of this method, inference context data consists of visual embeddings generated by an image encoder or a video encoder in a multimodal inference model. Staging refers to the process of gradually moving the visual embeddings from the CXL memory device to the accelerator's local memory for processing in conjunction with text token embeddings by the multimodal inference model. The multimodal inference model may handle both visual and text inputs. Visual embeddings generated by image and video encoders are large and may be pre-computed and stored in CXL memory, then gradually moved to the accelerator as a multimodal inference request is sent.

[0183] In some implementations of this method, the inference context data consists of pre-computed embeddings associated with retrieved documents within the Search Augmented Generation (RAG) pipeline. Staging involves staging the pre-computed embeddings from the CXL memory device to the accelerator's local memory to be concatenated with the input query during inference. In the RAG pipeline, retrieved document embeddings can be pre-computed and stored in the CXL memory device as a large-capacity embedding store. When a query triggers a search, the associated embeddings are incrementally moved from the CXL memory device to the accelerator and may be concatenated with the query token before the generation phase.

[0184] In some implementations of this technique, the inference context data consists of key-value (KV) cache entries related to token locations in the input sequence. Staging involves staging a subset from the CXL memory device to the accelerator's local memory, selected based on the token locations to be accessed in the next inference step. The remaining KV cache entries related to the token locations are then retained in the CXL memory device. In long-term context inference, the host stages only the KV cache entries related to token locations expected to be addressed in the next decode step, retaining the remaining entries in CXL memory. This selective staging reduces transfer volume and accelerates memory load while maintaining a complete context that is searchable on demand.

[0185] In some implementations of this method, an active cable is provided, consisting of first and second pluggable modules coupled in a physical medium, where the active cable further constitutes hardware circuitry, integrated into the active cable and configured to perform this method. In some implementations of this technique, a device acting as a switch is provided, which consists of switching circuitry and is configured to perform the method.

[0186] In an AI inference deployment, the system includes a cluster of accelerators connected via a UALink switch, one or more CXL memory devices providing large memory capacity, an RPU for translating between UPLI and CXL.mem, and a host for scheduling workloads and orchestrating data movement. Such a system can be configured to migrate inference context data across UALink and CXL protocol domain boundaries between the accelerator's local memory and the CXL memory device, enabling the aggregation of inference workloads that exceed the local memory capacity of a single accelerator. The host can access the CXL memory device via CXL.mem, and the accelerators can access the same CXL memory device through the RPU that translates between UPLI and CXL.mem. This architecture may support hierarchical memory management, where inference context data is incrementally distributed between the accelerator's HBM, HBF, host memory, and CXL memory based on access patterns, scheduling policies, and memory capacity constraints.

[0187] Various implementations offer the following systems: Ultra Accelerator Link (UALink) switches; accelerators are connected to the UALink switch, and each accelerator constitutes local memory. Compute Express Link (CXL) memory devices; resource provisioning units (RPUs) are connected to the UALink switch and CXL memory devices, and the RPUs are configured to translate between the UALink protocol-level interface (UPLI) and CXL.mem. Hosts connected to CXL memory devices via CXL.mem; in this system, inference context data is configured to migrate between the UALink protocol domain and the CXL protocol domain between the local memory of at least one accelerator and the CXL memory device. This system can enable both accelerator-driven and host-driven data migration across protocol domain boundaries. When communicating with accelerators and with the RPU, the host can communicate with the CXL memory device using CXL.mem via the UALink switch using UPLI. The RPU may be implemented as a discrete component connected to the UALink switch, as an IP block embedded within the accelerator, or as a chiplet within an IC package. The system supports a variety of inference workloads, and inference context data includes KV cache data, model weight parameters, activation tensors, embedding vectors, or other data generated or consumed during inference computations. CXL memory devices may include CXL memory expanders, CXL memory pools, or GFDs.

[0188] In some implementations of the system, at least one accelerator is configured to initiate the migration of inference context data by sending an UPLI request to the RPU via the UALink switch, which is then configured to translate the UPLI request into a CXL.mem M2S request and send the CXL.mem M2S request to the CXL memory device. The accelerator can determine when to migrate the inference context data based on local memory capacity utilization, access patterns, and workload scheduling signals received from the host or other accelerators.

[0189] In some system implementations, local memory constitutes the first memory layer, the entire system is coupled with host memory, the host memory constitutes the second memory layer, and the CXL memory device constitutes the third memory layer. Inference context data is then migrated between the first and third memory layers based on either access frequency, access recency, the capacity of the first memory layer, or the host's scheduling policy. The layered memory hierarchy allows the system to hold actively accessed inference context data in the highest bandwidth layer (accelerator local memory) while staging less frequently accessed data to a lower-cost, high-capacity layer (CXL memory). The second memory layer (host memory) may function as an intermediate staging buffer.

[0190] In some implementations, the system further comprises a second CXL memory device and a CXL fabric connecting the RPU to the CXL memory device and the second CXL memory device. At least one of the CXL memory devices, or the second CXL memory device, includes either a CXL memory expander, a CXL memory pool, or a global fabric-attached memory device (GFD). The CXL fabric may include one or more CXL switches that provide connectivity between the RPU and the CXL memory devices. The RPU can route translated requests to specific CXL memory devices based on the physical address contained in the UPLI request.

[0191] In some implementations of the system, inference context data consists of key-value (KV) cache data associated with the trans-based inference model, local memory consists of high-bandwidth memory (HBM), and at least one accelerator consists of either a graphics processing unit (GPU) or a tensor processing unit (TPU). GPUs and TPUs are accelerators that generate large amounts of KV cache data for trans-based inference workloads. HBM provides the high-bandwidth, low-latency access required for active inference computations, and CXL memory devices may provide additional capacity for KV cache entries that are not actively accessed.

[0192] In environments that communicate using NVLink-based protocols, such as GPUs coupled via NVLink or NVSwitches, it may be necessary to migrate inference context data to CXL memory devices residing in different protocol domains. The RPU translates between NVLink-based protocol messages and CXL.mem messages, enabling accelerators in the NVLink domain to stage the inference context data to the CXL memory device. The RPU can be integrated into an active cable, NVSwitch, or discrete bridge device and can perform address translation between NVLink and CXL.mem addresses, as well as command and tag translation between the two protocol domains. This NVLink-to-CXL.mem translation path can enable heterogeneous computing environments where different accelerator clusters utilize different interconnect protocols while sharing access to common CXL memory resources.

[0193] Various implementations include techniques for migrating inference context data across protocol domain boundaries. The details are as follows: an accelerator sends a request conforming to an NVLink-based protocol to a resource provisioning unit (RPU), the request related to the inference context data is stored in the local memory of the accelerator, and the request containing the first physical address is transmitted; the RPU converts the request into a Compute Express Link (CXL) CXL.mem Master-to-Subordinate (M2S) request containing a second physical address; and the RPU sends the CXL.mem M2S request to a CXL memory device; the inference context data is migrated between the local memory of the accelerator and the CXL memory device, across the NVLink protocol domain and the CXL protocol domain. This technique may allow accelerators that communicate according to the NVLink-based protocol to access CXL memory resources for staging inference context data. The RPU performs conversion between NVLink-based protocol messages and CXL.mem messages, including conversion of addresses, commands, and identifiers. The first physical address may be an NVLink network address, and the second physical address may be an HPA used in the CXL memory device. The RPU may be integrated into an active cable, an NVSwitch, or a discrete bridge device. In some examples, CXL operates over PCIe electrical circuits designed for short-range connections, and NVLink incorporates signal characteristics compatible with long-range interconnects, so the RPU may be positioned closer to the CXL port to optimize signal integrity.

[0194] In some implementations of this method, the accelerator is connected to the RPU via an NVLink switch (NVSwitch), and requests compliant with NVLink-based protocols are sent to the RPU through the NVSwitch. The NVSwitch can route NVLink-based requests from the accelerator to the RPU based on the routing information contained in the request. The NVSwitch connects the accelerator to the RPU, enabling the accelerator in the NVLink domain to access the CXL memory device to stage inference context data.

[0195] In some implementations of this method, the inference context data consists of key-value (KV) cache data generated during attentional computations in the trans-based inference model, and the local memory consists of high-bandwidth memory (HBM). KV cache data generated by the NVLink coupled accelerator during trans-based inference may be gradually migrated to a CXL memory device when the KV cache exceeds the HBM capacity.

[0196] Figure 13 shows an example system consisting of an accelerator cluster (e.g., GPUs and TPUs) interconnected by an UALink switch, an RPU connected to the UALink switch, a CXL memory device connected to the RPU via a CXL fabric, and a host coupled to the CXL memory device via CXL.mem. The path from Path 1a to Path 1b represents the data migration path by the accelerator. Accelerator.2 sends an UPLI request to RPU.p through the UALink switch. RPU.p translates the UPLI request into a CXL.mem M2S request and sends the CXL.mem M2S request to the GFD through the CXL fabric. The RPU performs address translation between the UALink address space (e.g., NPA and SPA) and the CXL address space (e.g., HPA), and can also translate opcodes, tags, and additional fields between the OPLI and CXL.mem message formats. In the write direction, the accelerator can evicte inference context data (such as HBM and HBF) from local memory to the GFD by sending a UPLI write request over the originator data channel. In the read direction, the accelerator can retrieve inference context data from the GFD by sending a UPLI read request, and the RPU receives CXL.mem S2M DRS from the GFD, converts CXL.mem S2M DRS to UPLI RdRsp, and returns the data to the accelerator via the UALink switch.

[0197] The path from path 2a to 2b represents a host-initiated access path. The host accesses the GFD via CXL.mem without translation by the RPU. The host can read inference context data from the GFD via CXL.mem and then write the inference context data to the accelerator by sending a CXL.mem M2S RwD to the RPU. The RPU translates the CXL.mem M2S request into a UPLI request and then sends the UPLI request to the accelerator via the UALink switch. Conversely, the host can read the inference context data by sending a CXL.mem M2S request to the RPU, which translates it into a UPLI read request, receives the UPLI RdRsp from the accelerator, and returns that data to the host. The host can write data to the GFD via CXL.mem. The host can orchestrate bidirectional data staging across the boundaries of the CXL and UALink protocol domains between the CXL memory device and the accelerator.

[0198] For example, the host may bring inference context data, such as KV cache data, from storage (e.g., NVMe or remote storage) to a CXL memory device such as a GFD. The accelerator can then retrieve the KV cache data from the GFD to local memory (e.g., HBM or HBF) and perform active inference computations. If the KV cache data is no longer actively accessed or if the local memory capacity is exceeded, the accelerator may migrate the cold KV cache data from the HBM to the GFD. In this way, the system maintains a hierarchical memory hierarchy, with the accelerator HBM acting as the working memory layer for actively accessed inference context data, and the CXL memory device acting as the capacity layer for less frequently accessed inference context data. The RPU may enable data migration between these layers by translating across protocol domain boundaries between UPLI and CXL.mem.

[0199] The CXL memory devices shown in Figure 13 may include CXL memory expanders, CXL memory pools, or GFDs, and may be connected to the RPU via a CXL fabric. The CXL fabric may include one or more CXL switches. The RPU may route translated requests to specific CXL memory devices based on the physical address embedded in the translated CXL.mem M2S requests. The system may support various types of inference context data, such as KV cache data related to trans-based models, model weight parameters related to expert hybrid models, activation tensors, embedding vectors, and other data generated or consumed during inference computations.

[0200] In environments where entities using different protocols require access to shared memory resources, the first entity communicating according to UPLI (such as an accelerator or UALink switch) may need to access memory resources located in the CXL domain (e.g., host memory accessed via a CXL host). RPUs and other appropriate devices may perform translations between UPLI and CXL.cache to facilitate memory operations across different protocol domains. This translation includes converting UPLI requests containing read / write commands into CXL.cache D2H requests containing the corresponding read / write opcodes, and converting CXL.cache H2D data messages and responses into UPLI read responses. It also includes conversions of addresses, tags, CQIDs, and data formats.

[0201] Various implementations employ the following methods: The first entity receives an Ultra Accelerator Link Protocol Level Interface (UPLI) request consisting of read commands; the Resource Provisioning Unit (RPU) translates the UPLI request into a CXL.cache device-to-host (D2H) request containing read opcodes, where CXL indicates a Compute Express link; the D2H request for CXL.cache is sent to a second entity; the second entity receives a device (H2D) data message from the CXL.cache host, the RPU translates the H2D data message for CXL.cache into a UPLI read response, and the UPLI read response is sent to the first entity. This method is translated between the UPLI domain and the CXL.cache domain. The RPU receives UPLI requests containing read commands, AtomicR commands, read commands returned via vendor-defined command (Rad RSP / Data) channels, or other read command types supported by UPLI. The RPU translates these read commands into corresponding CXL.cache read opcodes such as RdCurr, RdOwn, RdShared, or RdAny. This translation adapts operations to the requirements and capabilities of that domain. Upon receiving a data response from a CXL host, the RPU translates the data for the caller, abstracting the differences between the two protocols while enabling communication between the UPLI and CXL domains.

[0202] In some implementations of this method, a UPLI request consists of a source accelerator identifier and a destination accelerator identifier, and a UPLI read response consists of a destination accelerator identifier with a value matching the source accelerator identifier of the UPLI request. The RPU stores at least the source accelerator identifier of the UPLI request and uses it to generate the UPLI read response. In some examples, the UPLI request includes identifiers that specify both the destination and destination accelerators. The RPU stores at least the source accelerator identifier when it receives a UPLI request. When generating a UPLI read response, the RPU uses the source accelerator identifier stored from the request as the destination accelerator identifier for the response and routes the message appropriately to the original requester. In some examples, the RPU stores the destination accelerator identifier of the UPLI request and may include the source accelerator identifier in the UPLI read response for debugging purposes, but this is not necessarily required for basic functionality.

[0203] In some implementations of this technique, a UPLI request further consists of an initial address and an initial tag, a D2H request for CXL.cache consists of a second address translated from the initial address and a command queue identifier (CQID) associated with the initial tag, and a UPLI read response constitutes the initial tag. In some examples, the RPU performs address translation between physical addresses in the UALink address space and physical addresses in the CXL.cache address space, maintaining the correlation between requests and responses using Tag / CQID mapping. The RPU translates UALink addresses to corresponding CXL.cache addresses based on the configured address mapping. Furthermore, the RPU maintains a mapping between UPLI tags and CXL.cache CQIDs, appropriately relating responses to the original requests. This mapping allows the RPU to include the original UPLI tag in UPLI read responses, enabling the caller to associate the response with the pending request.

[0204] In some implementations of this technique, UPLI requests also include a length field indicating the amount of data requested. Furthermore, translating a UPLI request means generating multiple CXL.cache D2H requests if the requested data exceeds the maximum CXL.cache transaction size. The RPU may process UPLI requests that specify data transfers exceeding what a CXL.cache transaction can support. The length field of a UPLI request indicates the amount of data requested, up to 256 bytes, while CXL.cache transactions typically handle up to 64 bytes each. If the requested data size exceeds the CXL.cache transaction size limit, the RPU may split the request into multiple CXL.cache transactions. For example, a 256-byte UPLI request might be translated into four 64-byte CXL.cache transactions, with the RPU managing these multiple transactions while making it appear to the caller as a single operation.

[0205] In some implementations of this method, UPLI requests are composed of tags, and D2H requests for CXL.cache are composed of command queue identifiers (CQIDs). Furthermore, when generating multiple CXL.cache D2H requests, this involves associating multiple CQIDs with a tag. When dealing with large-scale data transfers requiring multiple CXL.cache transactions, the RPU may implement a tag-to-CQID mapping. For a UPLI consisting of a single tag, the RPU can generate and track multiple CQIDs for each CXL.cache transaction. For example, if a UPLI request containing tag #1 requires four CXL.cache transactions, the RPU can generate CQIDs #1 through #4 and maintain a mapping between the CQIDs and the original tag #1. This mapping allows the RPU to associate the relevant CXL.cache responses with the original UPLI request and generate a unified response.

[0206] In some implementations of this technique, the conversion of CXL.cache H2D data messages further involves receiving multiple CXL.cache H2D data messages corresponding to multiple CXL.cache H2D data messages, aggregating from the multiple CXL.cache H2D data messages, and including the aggregated data in the UPLI read response. When handling large data transfers split across multiple CXL.cache transactions, the RPU may perform data aggregation before generating the UPLI read response. The RPU receives multiple CXL.cache H2D data messages, collects those containing the requested data, and reassembles them as needed in a predetermined order to form a complete data payload. The RPU manages this aggregation process while maintaining data integrity and proper ordering, ensuring that the sender receives a single, consistent response containing the requested data.

[0207] In some implementations of this technique, the read opcode for a CXL.cache D2H request is RdCurr, and processing a CXL.cache D2H request does not modify the existing state of the cache accessible by the RPU. In some examples, the RPU translates a UPLI read command into a CXL.cache D2H request consisting of RdCurr, indicating that the memory read is performed by a read function of the current state. This means that the RPU does not issue cache commands, or that processing a CXL.cache RdCurr request does not modify the existing state of the cache accessible by the RPU.

[0208] In some implementations of this method, the read command is selected from at least one of the following: a read command, an AtomicR command, or a vendor-defined command, and the read data is returned through the UPLI read response / data (Rd Rsp / Data) channel. Additionally, the read opcode is selected from at least one of RdCurr, RdOwn, RdShared, or RdAny. The RPU supports conversions between various read commands within the UPLI domain and may support read opcodes from the CXL domain. UPLI read commands may include one or more vendor-defined read commands for memory reads, atomic operations providing atomicity guarantees, and custom functions. UPLI read commands are converted to the appropriate CXL.cache read opcode based on the requirements of the operation. For example, RdCurr requests the current value without changing the cache state, RdOwn acquires exclusive ownership, RdShared enables shared access, and RdAny allows flexible coherency state transitions.

[0209] In some implementations of this method, a CXL.cache H2D data message consists of the first data in the first format. Translating a CXL.cache H2D data message means converting the first data into a second data in a second format for UPLI read responses. The RPU may need to convert data between different formats when moving between the UPLI domain and the CXL.cache domain, which may involve changes in data organization, alignment, and representation. The RPU can maintain data integrity while performing the relevant format conversions to meet protocol requirements. The former is optimized for CXL.cache operations, while the second format conforms to the UPLI specification for data transfer.

[0210] In some implementations, this method also receives a UPLI request from the first entity containing the write command and the data to write. The RPU converts the UPLI request containing the write command into a CXL.cache D2H request containing a write opcode selected from DirtyEvict, ItoMWr, WrCur, or WrInv. It then sends the CXL.cache D2H request, consisting of the write opcode and the data to write, to the second entity, where the first entity consists of a UPLI originator, accelerator, or UALink switch. If the second entity consists of a CXL host, the RPU consists of a CXL device and exposes the CXL endpoint to the second entity. The RPU can convert the UPLI write command into a CXL.cache D2H write opcode. DirtyEvict is used to evict modified cache lines from the RPU, ItoMWr is used for write operations that upgrade cache lines from invalid to modified, WrCur writes to cache lines on cache hits and to memory on cache misses, and WrInv can write data while invalidating cache lines. In some examples, the first entity may be the UPLI originator (such as an accelerator or UALink switch) that initiates the write transaction, and the second entity may be the CXL host that receives the translated CXL.cache D2H write request. The RPU may expose CXL endpoints, such as Type-1 or Type-2 CXL devices, to CXL hosts and engage in CXL.cache communication.

[0211] In some implementations of this method, an active cable is provided, consisting of first and second pluggable modules coupled in a physical medium, where the active cable further constitutes hardware circuitry, integrated into the active cable and configured to perform this method. In some implementations of this technique, a device acting as a switch is provided, which consists of switching circuitry and is configured to perform the method.

[0212] The device includes a first interface configured for UPLI-compliant communication and a second interface configured for communication according to CXL.cache, with the computer connected to both interfaces and configured to translate between the two protocol domains. The device may be implemented as a discrete component, as an IP block embedded within an accelerator or processor, as a chiplet within an IC package, or integrated into an active cable. The computer maintains the mapping between UPLI tags and CXL.cache CQIDs, performs address translation between UALink and the CXL address space, and aggregates data from multiple CXL.cache transactions if a UPLI request exceeds the maximum CXL.cache transaction size. The first interface may include a UPLI confiber that receives UPLI requests from UPLI callers.

[0213] In various implementations, a device such as the following includes: a first interface configured to communicate according to the Ultra Accelerator Link Protocol Level Interface (UPLI); and a second interface configured to communicate according to CXL.cache, where CXL stands for Compute Express Link. A computer connected to the first and second interfaces is configured to translate UPLI requests containing read commands received on the first interface into CXL.cache device-to-host (D2H) requests containing read opcodes, and to send CXL.cache D2H requests via the second interface. It also translates CXL.cache host-to-device (H2D) data messages into UPLI read responses via the second interface and sends UPLI read responses via the first interface.

[0214] In some implementations of the device, a UPLI request consists of a source accelerator identifier and a destination accelerator identifier, and a UPLI read response consists of a destination accelerator identifier with a value matching the source accelerator identifier of the UPLI request. Additionally, the computer is configured to at least store the source accelerator identifier of the UPLI request and use it to generate the UPLI read response. In some examples, the computer processes the identifier information embedded in the UPLI message, stores at least the source accelerator identifier from the UPLI request, and uses the stored source accelerator identifier as the destination accelerator identifier in the response to appropriately route the message back to the original requester and generate the UPLI read response. Optionally, the destination accelerator identifier of the UPLI request can be stored and included in the UPLI read response for debugging purposes, but this is not necessarily required for basic functionality.

[0215] In some implementations of the device, a UPLI request further consists of an initial address and an initial tag, a CXL.cache D2H request consists of a second address translated from the initial address and a command queue identifier (CQID) associated with the initial tag, and a UPLI read response consists of the initial tag. Furthermore, the computer is configured to receive a CXL.cache H2D response through a second interface, consisting of a global observation (GO) opcode and a GO opcode indicating the cache coherency state. In some examples, address translation is performed between physical addresses in the UALink address space and physical addresses in the CXL.cache address space, and the correlation between requests and responses is maintained using Tag / CQID mapping. Based on the configured address mapping, the computer translates the UALink address to the corresponding CXL.cache address. Furthermore, it maintains a mapping between the UPLI tag and the CXL.cache CQID to appropriately associate the response with the original request. This mapping allows the computer to include the original UPLI tag in the UPLI read response, and the caller to associate the response with the pending request.

[0216] In some implementations of the device, a UPLI request further consists of a length field indicating the amount of data requested. Furthermore, if the requested data exceeds the maximum CXL.cache transaction size, the computer is further configured to generate multiple CXL.cache D2H requests. The computer may process UPLI requests that specify a data transfer greater than a single CXL.cache transaction can support. The length field of a UPLI request indicates the amount of data requested, up to 256 bytes, while CXL.cache transactions typically handle up to 64 bytes each. If the requested data size exceeds the CXL.cache transaction size limit, the computer may split the request into multiple CXL.cache transactions. For example, a 256-byte UPLI request might be translated into four 64-byte CXL.cache transactions, and the computer manages these multiple transactions while making it appear to the caller as a single operation.

[0217] In some implementations of the device, UPLI requests consist of tags, and D2H requests for CXL.cache consist of command queue identifiers (CQIDs). Furthermore, the computer is configured to associate multiple CQIDs with tags when generating multiple CXL.cache D2H requests. When handling large data transfers that require multiple CXL.cache transactions, the computer may implement a tag-to-CQID mapping. For a UPLI request consisting of a single tag, the computer can generate and track multiple CQIDs, one for each CXL.cache transaction. This mapping allows the associated CXL.cache responses to be associated with the original UPLI request, enabling the generation of a unified response.

[0218] In some implementations of the device, the computer is further configured to: receive multiple CXL.cache H2D data messages in response to multiple CXL.cache D2H requests, aggregate data from multiple CXL.cache H2D data messages, and include the aggregated data in the UPLI read response. When handling large data transfers split across multiple CXL.cache transactions, the computer may perform data aggregation before generating the UPLI read response. The computer receives multiple CXL.cache H2D data messages, collects those containing the requested data, and reassembles them as needed in a predetermined order to form a complete data payload. The computer manages this aggregation process while maintaining data integrity and proper order, so that the sender receives a single, consistent response containing the requested data.

[0219] In some implementations of the device, the read opcode for a CXL.cache D2H request is RdCurr, and the computer is further configured not to modify the existing state of the accessible cache as a result of processing the CXL.cache D2H request. Some examples translate the UPLI read command into a CXL.cache D2H request consisting of RdCurr, indicating that the memory read is performed by a read function of the current state. In some examples, the device does not issue a cache command, or when processing a CXL.cache RdCurr request, the device retrieves the data in a way that does not change the state of the accessible cache.

[0220] In some implementations of the device, the read command is selected from at least one: the Read command, the AtomicR command, or a vendor-defined command that returns read data on the UPLI read response / data channel (Rd Rsp / Data); and the read opcode is selected from at least one of RdCurr, RdOwn, RdShared, or RdAny. The computer supports conversion between various read commands within the UPLI domain and can support read opcodes in the CXL domain. UPLI read commands may include one or more vendor-defined read commands for memory reads, atomic operations that provide atomicity guarantees, and custom functions. UPLI read commands are converted to the appropriate CXL.cache read opcode based on the requirements of the operation. For example, RdCurr requests the current value without changing the cache state, RdOwn acquires exclusive ownership, RdShared enables shared access, and RdAny enables flexible coherency state transitions.

[0221] In some implementations of the device, CXL.cache H2D data messages contain data in the first format. Furthermore, the computer is further configured to convert the first data to a second format for UPLI read responses. The computer may need to convert data between different formats when moving between the UPLI domain and the CXL.cache domain, which may involve data reorganization, alignment, and changes in representation. The computer can maintain data integrity while performing the relevant format conversions to meet protocol requirements. The former is optimized for CXL.cache operations, while the second format performs data transfer in accordance with the UPLI specification.

[0222] In some implementations of the device, the computer is further configured as follows: it converts a UPLI request containing write commands and write data received through the first interface into a CXL.cache D2H request containing a write opcode selected from DirtyEvict, ItoMWr, WrCur, or WrInv. It then sends the CXL.cache D2H request, consisting of the write opcode and write data, through a second interface. Here, the first interface consists of a UPLI compiler. The computer can convert UPLI write commands into CXL.cache and D2H write opcodes. The first interface may include a UPLI confeller, which can receive UPLI requests from UPLI originators such as accelerators or UALink switches.

[0223] In some implementations of the device, the device consists of a chiplet within the processor's integrated circuit (IC) package, and the processor consists of a coherent interconnect coupling processing core to the memory controller. If a second interface connects to the coherent interconnect via a CXL root port, this allows the computer to translate UPLI requests into CXL.cache D2H requests and access memory in conjunction with the memory controller via the coherent interconnect. This device may be implemented as a chiplet within the IC package of a processor, such as an MxPU. Coherent interconnects, such as ring or mesh interconnects, may couple the processing core, LLC, CXL RP, and memory controller. The computer may use CXL.cache as an intermediate protocol to translate between UPLI and the protocols used by the coherent interconnect. The CXL root port may be included in the chiplet or it may be part of the processor, communicating with the coherent interconnect from the ring via a CXL (R2CXL) interconnect interface.

[0224] In some implementations of the device, an active cable is connected to a first pluggable module connected to a first physical medium, and a second pluggable module is connected to a second physical medium, thereby configuring a device that combines the first and second physical media according to claim 13.

[0225] In an AI inference environment, accelerators may run neural network models where the aggregate data requirements exceed the accelerator's local memory capacity. Model data such as expert weights, KV cache entries, and attention parameters may be stored in host memory and retrieved on demand during inference. The RPU can perform conversions between UPLI used by the accelerator and CXL.cache used by the host, allowing the accelerator to read model data from host memory and write inference context data back to host memory across protocol domain boundaries.

[0226] In various implementations, the following approach is used: an accelerator running in local memory executes an inference workload on a neural network model; the accelerator sends an Ultra Accelerator Link Protocol Level Interface (UPLI) request containing a command to read model data stored in host memory; the Resource Provisioning Unit (RPU) translates the UPLI request into a CXL.cache device-to-host (D2H) request containing a read opcode, where CXL indicates a Compute Express link; the RPU sends the CXL.cache D2H request to the host; the RPU receives a CXL.cache host-to-device (H2D) data message containing the model data from the host; the RPU translates the CXL.cache H2D data message into a UPLI read response containing the model data; and the RPU sends the UPLI read response to the accelerator. This approach allows the accelerator running the inference workload to access model data stored in host memory via the RPU. The RPU translates the UPLI domain used by the accelerator and the CXL.cache domain used by the host, allowing the accelerator to expand the memory capacity it can access beyond local memory. Model data includes neural network weights, activation data, intermediate computation results, or other data related to the inference process. Host memory can provide a larger capacity than the accelerator's local memory, and the RPU bridges the two protocol domains to enable memory capacity scaling for inference workloads.

[0227] In some implementations of this technique, the model data consists of at least one: expert weights associated with a Mixture-of-Expert (MoE) model, which are retrieved from host memory when the gate network of the MoE model routes one or more tokens to experts; key-value (KV) cache entries offloaded from local memory to host memory during sequence inference; or attention parameters associated with a transformer-based model. In a MoE model, only a portion of the expert subnetwork may be active for any given input token. Weights of inactive experts are offloaded to host memory, freeing up local memory capacity for accelerators or retrieving activated expense weights from host memory when the gate network routes tokens to experts. KV cache entries may also be offloaded to host memory during long sequence inference, freeing up local memory for active computations and fetched back for attention computations as needed. This dynamic staging may enable model inference beyond the available local memory capacity.

[0228] In some implementations, this technique further involves the accelerator sending a UPLI request containing write commands and inference context data. The RPU translates the UPLI request containing the write commands into a CXL.cache D2H request containing the write opcode. The CXL.cache D2H request and inference context data are sent to the host and stored in host memory. Depending on the write direction, the accelerator may offload inference context data to host memory, such as KV cache entries that are not actively needed, intermediate calculation results, and checkpoint data. The RPU may also translate the UPLI write command into a CXL.cache D2H write opcode such as DirtyEvict, ItoMWr, WrCur, or WrInv, and transfer and store the inference context data in host memory.

[0229] The system may include an accelerator with local memory, a host with host memory, and an RPU coupled between them, bridging the UPLI and CXL.cache protocol domains. The RPU allows the accelerator to access host memory capacity beyond its local memory, eliminating memory capacity limitations in workloads such as AI inference and high-performance computing. Host memory may include removable DDR modules or CXL-attached memory, providing a larger but higher latency capacity compared to the accelerator's local HBM. The UALink switch connects the accelerator to the RPU, enabling accelerators to share access to host memory via the RPU.

[0230] In various implementations, the following systems are included: an accelerator configured with local memory and configured to communicate according to the Ultra Accelerator Link Protocol Level Interface (UPLI); a host configured with host memory and configured to communicate according to CXL.cache, where CXL stands for Compute Express Link; and a Resource Provisioning Unit (RPU) connecting the accelerator and the host. The RPU consists of a first interface configured to communicate with the accelerator according to UPLI and a second interface configured to communicate with the host according to CXL.cache. Here, the RPU is configured as follows: it translates a UPLI request containing a read command received from the accelerator via the first interface into a CXL.cache device-to-host (D2H) request containing a read opcode, and sends the CXL.cache D2H request to the host via the second interface. It translates a CXL.cache host-to-device (H2D) data message received from the host via the second interface into a UPLI read response, and sends that UPLI read response to the accelerator via the first interface. This allows the accelerator to access host memory via the RPU. The system enables the accelerator to access host memory through protocol translation by the RPU. The RPU bridges the accelerator's UPLI domain with the host's CXL.cache domain, translating read commands into D2H read opcodes in CXL.cache and H2D data messages in CXL.cache into read responses in UPLI. The system can address the accelerator's memory capacity limitations by enabling access to host memory, which may offer a larger capacity than the accelerator's local memory. The RPU may be a standalone component, an IP block embedded in the processor, or a chiplet within an IC package.

[0231] In some system implementations, the host memory capacity may exceed the local memory capacity. Local memory consists of high-bandwidth memory (HBM), and host memory consists of removable DDR modules, i.e., CXL-connected memory. The accelerator accesses the combined memory capacity of local and host memory. Furthermore, the RPU is configured as follows: it converts a UPLI request containing the write command and write data received from the accelerator into a CXL.cache D2H request containing the write opcode. It then sends the CXL.cache D2H request, consisting of the write opcode and write data, to the host memory for storage. As an example, a removable DDR module refers to a standardized hardware assembly consisting of one or more memory integrated circuits placed on a board, including dual in-line memory modules (DIMMs), compressed-connected memory modules (CAMM or CAMM2), and all forms and mounting architectures of future modular memory form factors.

[0232] Some implementations also include an UALink switch that connects the accelerator and the RPU. The UALink switch routes UPLI requests from the accelerator to the RPU based on the destination accelerator identifier included in the UPLI request. The accelerator is also connected to the UALink switch and shares access to host memory via the RPU. The UALink switch routes UPLI traffic based on the destination accelerator identifier (e.g., the ReqDstPhysAccID field) and uses a route table to map the destination identifier to the exit port connected to the RPU. The accelerator can share access to host memory through the RPU, and the accelerator can expand the amount of memory it can access. The UALink switch may also route UPLI responses from the RPU to the outgoing accelerator based on the destination accelerator identifier.

[0233] Figure 14A shows an example of a system consisting of computers, which are contained within a switch or bridge and consist of a first interface (Interface.1) and a second interface (Interface.2). The first interface may communicate with the first entity (Entity.1), such as a CPU or accelerator, following an UALink-based protocol such as UPLI. The second interface may communicate with the second entity (Entity.2), such as a host, following CXL.cache. The computer can extract addresses from requests received through the first interface, and these addresses may refer to the first address space, such as the network physical address (NPA) space used by the first entity. The computer can further translate these addresses and generate requests with the translated addresses to send via the second interface. These translated addresses may refer to the second address space used by the second entity. In some examples, the first and second address spaces may be associated with the same address space, such as a common address space, global address space, pod address space, or fabric address space, and the computer may perform address translation between addresses within the same common address space. In other examples, it may be implemented in an IC package with high-speed differential I / O balls arranged according to a ball grid array layout defined in the retimer specification.

[0234] Figure 14B shows an example where TFD demonstrates translations such as protocol translation. This is a translation between an UALink-based request, such as a UPLI request received from the first entity (Entity.1), and a CXL.cache D2H request sent to a second entity (Entity.2), which may be the host. This allows the first entity to access resources mapped to the address space used by the second entity. The first entity can initiate a UPLI transaction, which includes a UPLI request containing ReqCmd (Read), ReqAddr (AS.4.1), and ReqTag (c.4.1). The computer converts the UPLI transaction into a CXL transaction, and can send the D2H request for CXL.cache (which may include Opcode(RdCurr), CQID(q.2.1), and Address(AS.2.1)) to a second entity. Upon receiving one or more responses from the second entity, the computer converts one or more of the responses into CXL.cache H2D data (which may include CQID(q.2.1) and Data(*Data.2*). For example, it converts the H2D data in CXL.cache into a UPLI read response / data (RdRsp), which consists of RdRspTag(c.4.1) and RdRspData(*Data.2*)), and then converts the UPLI The RdRsp is sent to the first entity. In some examples, the computer may perform additional translations, such as address translations sent in the CXL request sent to the second entity, such as address translations sent to the second entity, such as address translations sent to the second entity, such as address translations sent to the second physical address (such as address translations sent to the second physical address (HPA)), which is the address included in the UPLI request received from the first entity, such as the first physical address (such as the network physical address (NPA)).

[0235] The computer can perform further conversions such as opcode conversion, command conversion, TLP type conversion, and field conversion between UALink-based and CXL domains. In some examples, when the computer receives an UPLI request from the first entity, it may issue multiple CXL.cache D2H requests. For example, it might split a large data block (e.g., 256B) UPLI request into smaller CXL.cache D2H requests (e.g., 64B), or use CXL.cache reads to prefetch data from a second entity. The computer can convert requests and transactions initiated from UALink-based domains to CXL domains, requests and transactions initiated from CXL domains to UALink-based domains, and requests and transactions initiated from both domains.

[0236] Figure 14C shows an example where TFD demonstrates a translation similar to protocol translation. This is a translation between an UALink-based request (such as an UPLI request) received from the first entity (Entity.1, which may be a CPU or accelerator) and a CXL request sent to a CXL.cache request (which may be a host, and may not be a host), such as a CXL.cache D2H request. This allows the first entity to access resources mapped to the address space used by the second entity. The first entity can initiate a UPLI transaction containing ReqCmd(Read), ReqAddr(AS.3.1), and ReqTag(c.3.1). The computer translates the UPLI transaction into a CXL transaction, which may contain a D2H request for CXL.cache (Opcode(RdShared), CQID(q.1.1), and Address(AS.1.1)), and can send the D2H request for CXL.cache to the second entity. Upon receiving one or more responses from the second entity, this may include a CXL.cache H2D Rsp(Opcode(GO), RspData(S), CQID(q.1.1) may be included, and may also include H2D data of CXL.cache(CQID(q.1.1) and Data(*Data.1*). The computer translates one or more responses, for example, translating H2D data of CXL.cache into UPLI read response / data (RdRsp) (RdRspTag(c.3.1) and RdRspData(*Data.1*)). The RdRsp of UPLI can be sent to the first agency. In some cases, the computer may include a cache or control an external cache, in which case it may be configured to select the opcode for the CXL.cache D2H request according to the cache line state intent and cache policy identified based on the UPLI request (e.g., selecting RdShared to store a shared copy of the cache line in the cache).Alternatively, you might choose RdOwn to store an exclusive copy of the cache line in the cache.

[0237] In some cases, a computer may perform additional translations, such as address translation between the first physical address (e.g., the network physical address (NPA) included in the UPLI request received from the first entity) and the second physical address (e.g., the host physical address (HPA) included in the CXL request sent to the second entity). A computer can perform further translations, such as opcode translation, command translation, TLP type translation, and field translation between UALink-based and CXL domains. In some cases, a computer may issue multiple CXL.cache D2H requests when it receives a UPLI request from the first entity. For example, it might split a large data block (e.g., 256B) UPLI request into smaller CXL.cache D2H requests (e.g., 64B), or use CXL.cache reads to prefetch data from the second entity. A computer can translate requests and transactions initiated from UALink-based domains to the CXL domain, requests and transactions initiated from the CXL domain to the UALink-based domain, or requests and transactions initiated from both domains.

[0238] Figure 15A shows an example of a system composed of computers, which includes switches and bridges and consists of a first interface (Interface.1) and a second interface (Interface.2). The first interface may communicate with the first entity (Entity.1), which is a CPU or accelerator, following an UALink-based protocol such as UPLI. The second interface may communicate with switches and CXL devices (such as CXL memory expanders, CXL memory pools, GFDs, and CXL accelerators) following CXL.cache. The computer can extract addresses from requests received through the first interface, and these addresses may refer to the first address space, such as the network physical address (NPA) space used by the first entity. The computer can further translate these addresses and generate requests with the translated addresses to send via the second interface. These translated addresses may refer to the second address space used by the second entity. In some cases, the first and second address spaces may be associated with the same address space, such as a common address space, global address space, pod address space, or fabric address space, and the computer may perform address translation between addresses within the same common address space. In some cases, this may be implemented in an IC package with high-speed differential I / O balls arranged according to a ball grid array layout defined in the retimer specification.

[0239] Figure 15B shows an example of a TFD showing conversions such as protocol conversion. This is the conversion between an UALink-based request (such as a UPLI request) received from the first entity (Entity.1, which may be a CPU or accelerator) and a CXL request (either a switch or a CXL device), such as a CXL.cache H2D request. In some cases, this may allow the first entity to access resources related to the address space used by the second entity. The first entity can initiate a UPLI transaction, which may include a UPLI request containing ReqCmd (Read), ReqAddr (AS.4.1), and ReqTag (c.4.1). The computer converts the UPLI transaction into a CXL transaction, which may include a CXL.cache H2D request (Opcode (SnpCur), UQID (t.1.1), Address (AS.1.1)). You can also send an H2D request to a second entity. When you receive one or more responses from the second entity, these may include a D2H response in CXL.cache (Opcode(RspVFwdV) and UQID(t.1.1)), and possibly also a D2H data in CXL.cache (UQID(t.1.1) and Data(*Data.1*)). The computer may translate one or more of the responses. For example, it may translate the D2H data in CXL.cache into a UPLI read response / data (RdRsp) (RdRspTag(c.4.1) and RdRspData(*Data.1*)). The RdRsp in UPLI can then be sent to the first entity. In some examples, the computer may include or control an external cache, in which case it may be configured to select the opcode for the CXL.cache H2D request based on the intent of the cache line identified based on the UPLI request, or to select SnpData for the cache line that is to be cached, either in a shared or exclusive state.Alternatively, you can choose SnpCur to obtain the latest version of the cache line without changing the cache state within the hierarchy.

[0240] In some cases, a computer may perform additional translations, such as address translation between the first physical address (e.g., the network physical address (NPA) included in the UPLI request received from the first entity) and the second physical address (e.g., the host physical address (HPA) included in the CXL request sent to the second entity). A computer can perform further translations, such as opcode translation, command translation, TLP type translation, and field translation between UALink-based and CXL domains. In some cases, a computer may issue multiple CXL.cache H2D requests when it receives a UPLI request from the first entity. For example, it might split a large data block (e.g., 256B) UPLI request into smaller CXL.cache H2D requests (e.g., 64B), or use CXL.cache reads to prefetch data from the second entity. A computer can translate requests and transactions initiated from UALink-based domains to CXL domains, requests and transactions initiated from CXL domains to UALink-based domains, or requests and transactions initiated from both domains.

[0241] Figure 15C shows an example where TFD demonstrates a translation similar to protocol translation. This is the case when a UALink-based request (such as a UPLI request) is received from the first entity (Entity.1, which may be a CPU or accelerator), and a CXL request, such as a CXL.cache H2D request, is sent to the second entity (Entity.2), which may be a switch or CXL device. In some cases, the first entity may allow the second entity to access resources related to the address space used by the second entity. The first entity can initiate a UPLI transaction containing ReqCmd (Read), ReqAddr (AS.3.1), and ReqTag (c.3.1). The computer translates the UPLI transaction into a CXL transaction, which may contain a CXL.cache H2D request (Opcode (SnpData), UQID (t.2.1), Address (AS.2.1)), and CXL.cache An H2D request can be sent to a second entity. Upon receiving one or more responses from the second entity, this may include a D2H response in CXL.cache (Opcode(RspSFwdM) and UQID(t.2.1)), and possibly also a D2H data in CXL.cache (UQID(t.2.1) and Data(*Data.2*)). The computer translates one or more of the responses, for example, translating the D2H data in CXL.cache into a UPLI read response / data (RdRsp) (RdRspTag(c.3.1) and RdRspData(*Data.2*)). The RdRsp in UPLI can then be sent to the first entity. In some examples, the computer may include or control an external cache, in which case it may be configured to select the opcode for the CXL.cache H2D request based on the intent of the cache line identified based on the UPLI request, or to select SnpData for the cache line that is to be cached, either in a shared or exclusive state.Alternatively, you can choose SnpCur to obtain the latest version of the cache line without changing the cache state within the hierarchy.

[0242] In some cases, a computer may perform additional translations, such as address translation between the first physical address (e.g., the network physical address (NPA) included in the UPLI request received from the first entity) and the second physical address (e.g., the host physical address (HPA) included in the CXL request sent to the second entity). A computer can perform further translations, such as opcode translation, command translation, TLP type translation, and field translation between UALink-based and CXL domains. In some cases, a computer may issue multiple CXL.cache H2D requests when it receives a UPLI request from the first entity. For example, it might split a large data block (e.g., 256B) UPLI request into smaller CXL.cache H2D requests (e.g., 64B), or use CXL.cache reads to prefetch data from the second entity. A computer can translate requests and transactions initiated from UALink-based domains to CXL domains, requests and transactions initiated from CXL domains to UALink-based domains, or requests and transactions initiated from both domains.

[0243] Figure 16A shows an example of a system where a processor (e.g., MxPU) includes processing cores, LLC, CXL RP, and memory controller (such as DRAM), and is connected to memory via a memory channel. The CXL RP can communicate with coherent interconnects such as ARM CHI, Intel UPI, and AMD Infinity Fabric via a RING-to-CXL (R2CXL) interconnect interface. The RPU, which may be included in the MxPU, performs physical address translation, allowing entities such as GPUs to access memory. According to UPLI, the MxPU can expose UALink ports to entities through the RPU and communicate with them. The RPU may perform further translations, for example, from UPLI to protocols used by coherent interconnects, and in doing so, the RPU may use intermediate protocols such as CXL (e.g., CXL.cache). The RPU can expose CXL devices that utilize CXL endpoints (CXL EPs), such as Type 1 and Type 2 CXL devices, to the processor through the CXL RP included in the RPU. Here, the R2CXL residing within the RPU can communicate with the coherent interconnect and complete the translation path from UPLI to the protocol used by the coherent interconnect. In some examples, the RPU, UALink port, and CXL device may be implemented within a chiplet in the IC package of the MxPU, or they may be implemented as a functional block on the same die as the processor's CXL RP, or they may be split across processor dies and chiplets.

[0244] Figure 16B shows an example of a TFD (Transaction Function Diagram) illustrating a UPLI request (Req) received from an entity such as a GPU or accelerator. Here, the RPU can translate the physical address (AS.2.1) contained in the UPLI request to the physical address (AS.1.1) used for memory access. The RPU may perform further translations, such as protocol translation from UPLI to the protocol used by the processor's coherent interconnect, and may utilize intermediate protocols such as CXL.cache. The RPU may perform further translations such as OPCODE translation and tag translation, such as translating transaction tags from UPLI tags to CQIDs. The CXL.cache request, containing the translated address (AS.1.1), is then sent to the CXL RP for further processing and retrieval, where data is retrieved from the LLC via the on-chip ring interconnect or from memory such as DRAM via memory channels. The data is then returned to the RPU via the on-chip ring interconnect, and the RPU provides the requesting entity with a UPLI read response / data (RdRsp).

[0245] Figure 17A shows an example of a system composed of cables, such as active cables, which can be converted between UALink-based traffic and CXL-based traffic (traffic of any of CXL.io, CXL.mem, or CXL.cache). The cable / RPU allows a first entity (Entity.1), which is an accelerator, GPU, CPU, UALink switch, UPLI originator, or resource consumer, to connect to a second entity (Entity.2), which may be a host, accessing resources such as memory through UALink-based ports, cable / RPUs, and CXL-based ports. This second entity may be a CPU, CXL switch, CXL device, memory expander, memory pool, or resource provider. The cable / RPU may include an RPU, which can communicate with the first entity according to UALink-based protocols such as UPLI and with the second entity according to CXL-based protocols (any of CXL.io, CXL.mem, or CXL.cache). In some cases, UALink-based protocols like UPLI may be associated with a primary address space such as the NPA space, while CXL-based protocols like CXL.mem may be associated with a secondary address space such as the System Physical Address (SPA) space or the Host Physical Address (HPA) space. In this case, the cable / RPU can perform address translation between the primary and secondary addresses, such as between addresses in the NPA space, within the SPA space, or between addresses in the HPA space.

[0246] In other examples, UALink-based protocols like UPLI and CXL-based protocols like CXL.mem may be associated with the same physical address space, such as the global address space, pod address space, or fabric address space. In this case, the cable / RPU can perform address translation between addresses within the same address space. The cable / RPU may also perform further translations such as opcode, command, and TLP translation, including translation between the Read command in an OPLI request and the Read opcode in a CXL.cache D2H request. The cable / RPU can also translate between messages compliant with UALink-based protocols and messages compliant with CXL-based protocols, translate tags, and translate error indicators such as data corruption and poison.

[0247] In some cases, a cable / RPU such as OFP copper cable, OFP-XD copper cable, OFP active optical cable (AOC), or OFP-XD AOC cable may contain an additional first module (module 1). This module may contain a frit-enabled UALink retimer, PAM4 retimer, or retimer PHY, and may contain a further second module (module 2). The RPU may also optionally be included in discrete components corresponding to the retimer form factor. The cable / RPU may comply with standard specifications, industry standards, implementation agreements (IA), or multi-source agreements (MSA). Examples include Octal Small Form Factor Pluggable Module (OSFP), Octal Small Form Factor eXtra Dense Pluggable Module (OSFP-XD), Quad Small Form-Factor Pluggable (QSFP), Quad Small Form-Factor Pluggable Double Density (QSFP-DD), and the SNIA SFF standard (e.g., SFF-8665). These include the Optical Internet Forum (OIF) Implementation Agreement or the Institute of Electrical and Electronics Engineers (IEEE) standards. The cable / RPU may also contain physical media such as copper wire or optical fiber. In some examples, the cable / RPU may connect to an UALink-based port via a first electrical connector (Electrical Connector.1) and then to a CXL-based port via a second electrical connector (Electrical Connector.2). On the other hand, in other examples, the cable / RPU may connect to an UALink-based port or a CXL-based port via an optical connector.

[0248] Figure 17B shows an example of a system consisting of an active optical cable. An active optical cable may connect a GPU, accelerator, compute element, host, CPU, MxPU, UALink switch, UPLI originator, or consumer to a second entity (Entity 2), such as a host, CPU, GPU, accelerator, CXL switch, compute element, MxPU, memory pool, or provider. In some examples, the first entity may also be connected to the first port (Port 1), for example, an UALink port, via the first electrical connector (Electrical Connector 1). The active optical cable consists of a first module (Module 1) containing the first optical physical media-dependent device (Optical PMD 1), and a second module (Module 2), which may consist of the second optical physical media-dependent device (Optical PMD 2) and an RPU. The first module is connected to the first electrical connector and can convert electrical signals from the first entity into optical signals for transmission over the optical fiber. The second module can convert the optical signals back into electrical signals and may also constitute an RPU. The RPU may be converted between UALink-based protocols such as UPLI and CXL-based protocols that convert at least one of CXL.io, CXL.mem, or CXL.cache. The second module is connected to the second entity via a second electrical connector (Electrical Connector.2), and the second entity may be connected to a second port (Port.2), such as a CXL port.

[0249] In some cases, the RPU within the active optical cable may perform address translation between addresses in the first address space (NPA space) associated with UALink-based protocols and addresses in the second address space (HPA space, etc.) associated with CXL-based protocols. The RPU can also perform opcode translation, command translation, tag translation, and translation of error indicators such as data corruption and poison. In some cases, the active optical cable may comply with standards, industry standards, implementation agreements, or multiple-source agreements. Examples include Octal Small Form Factor Plug-in Module (OSFP), Octal Small Form Factor eXtra Dense Pluggable Module (OSFP-XD), Quad Small Form-Factor Plug-in Double Density (QSFP), SNIA SFF standards, Optical Internetworking Forum (OIF) Implementation Agreements, or IEEE standards. In other cases, the active optical cable may utilize other available form factors or standards.

[0250] Figure 18A shows an example of a system that includes a third entity (Entity.3), such as an xPU, bridge, or third switch (Switch.3). The third entity includes a computer consisting of a first interface (Interface.1) and a second interface (Interface.2). The first interface may communicate with the first entity (Entity.1), the first switch (Switch.1), the CPU, or an accelerator, following an UALink-based protocol such as UPLI. The second interface may communicate with a second switch (Switch.2), a CXL host, or a CXL device (such as a CXL memory expander, a CXL-based memory pool, a GFD, or a CXL accelerator), following a CXL-based protocol such as CXL.io, CXL.mem, or CXL.cache. This computer may perform translations such as protocol translation between UALink-based and CXL-based protocols, transaction translation, protocol data unit (PDU) translation, message translation, or address translation between UALink-based and CXL-based protocol domains. In some cases, a computer may extract addresses from requests received via the first interface, and these addresses may refer to the first address space, such as the network physical address (NPA) space used by the first entity. The computer can translate these addresses and generate a request containing the translated addresses, which it can then send via the second interface. These translated addresses may refer to a second address space, such as the host physical address (HPA) space used by the second entity. In other cases, the first and second address spaces may be associated with the same address space, such as a common address space, global address space, pod address space, or fabric address space, and the computer may perform address translations between addresses within the same common address space.In some cases, the computer may be implemented as a chiplet, utilizing UCIe for inter-chip or inter-die connections, or in an IC package with high-speed differential I / O balls arranged according to a ball grid array layout defined by the retimer specification.

[0251] Figure 18B illustrates an example where TFD uses UALink-based protocols (such as UPLI) to perform translations between the first entity (Entity.1), the first switch (Switch.1), the CPU, or an accelerator, including protocol translation, transaction translation, protocol data unit (PDU) translation, message translation, and address translation. Then, CXL-based protocols such as CXL.io, CXL.mem, and CXL.cache are used by the second entity (Entity.2), such as the second switch (Switch.2), a CXL host, or a CXL device. The first entity can initiate a UPLI transaction containing a UPLI request, including ReqCmd(r.3.1), such as Read or Read class Vendor Defined Command, ReqSrcPhysAccID(id.1), ReqDstPhysAccID(id.2), ReqAddr(AS.3.1), and ReqTag(c.3.1). The computer converts a UPLI transaction into a CXL transaction, which may include a CXL request (TransactionID(p.2.1) and Address(AS.2.1)), and can then send the CXL request to a second entity. In some examples, transaction IDs such as CXL.io tags, CXL.mem tags, CXL.cache CQID, and CXL.cache UQID are used as transaction identifiers to match responses and completions against requests, and also to identify pending transactions in progress. When the computer receives a CXL response or completion (CXL response / completion) consisting of TransactionID(p.2.1) and *Data.1* from the second entity, it can convert the CXL response / completion into a UPLI read response / data consisting of RdRspDstPhysAccID(id.1), RdRspTag(c.3.1), and RdRspData(*Data.1*), and send the UPLI read response / data to the first entity.

[0252] Computers may also perform opcode conversion, command conversion, TLP type conversion, or other conversions between UALink-based and CXL-based protocol domains. In some cases, a computer may issue multiple CXL requests when it receives a UPLI request from the first entity. For example, a computer may issue CXL.mem M2S requests from a MemRd or MemRdData, for instance, to split a large data block (e.g., 256B) in a UPLI request into smaller CXL.mem reads (e.g., 64B), or to prefetch data from a second entity using a CXL.mem read. Computers can convert requests and transactions initiated from UALink-based protocol domains to CXL-based protocol domains, or vice versa.

[0253] Figure 18C shows examples of translations performed by the computer, such as protocol translation, transaction translation, protocol data unit (PDU) translation, message translation, and address translation, for TFDs. These are used by UALink-based protocols like UPLI, the first entity (Entity.1, Switch.1), the CPU, or an accelerator. Then, they are used by second entities (Entity.2), such as the second switch (Switch.2), CXL host, or CXL device, for CXL-based protocols like CXL.io, CXL.mem, and CXL.cache. The first entity can initiate a UPLI transaction, which includes a UPLI request containing a Write or Write class Vendor Defined Command, ReqSrcPhysAccID(id.1), ReqDstPhysAccID(id.2), ReqAddr(AS.4.1), ReqTag(c.4.1), etc. The UPLI transaction may further include UPLI OrigData, which contains *Data.2*. The computer can convert a UPLI transaction into a CXL transaction and send a CXL request containing TransactionID(p.2.1), Address(AS.2.1), and *Data.2*. In some examples, transaction IDs such as CXL.io tags, CXL.mem tags, CXL.cache CQID, and CXL.cache UQID are used as transaction identifiers to match responses and completions against requests, and also to identify pending transactions in progress. If the computer receives a CXL response or completion (CXL response / completion) containing TransactionID(p.2.1) from the second entity, it can convert the CXL response / completion into a UPLI write response consisting of WrRspDstPhysAccID(id.1), WrRspTag(c.4.1), and WrRspStatus(d.4.1), and send the UPLI write response to the first entity.

[0254] Computers may also perform opcode conversion, command conversion, TLP type conversion, or other conversions between UALink-based and CXL-based protocol domains. In some cases, a computer may issue a CXL request when it receives a UPLI request from the initial entity. For example, a computer may issue multiple CXL.mem M2S RwDs (MemWr or MemWrTEE). For instance, a UPLI request might split a large data block (e.g., 256B) into smaller CXL.mem writes (e.g., 64B). Computers can convert requests and transactions initiated from UALink-based protocol domains to CXL-based protocol domains, or vice versa.

[0255] In computing environments where entities communicating according to CXL need to access data and memory resources in the UALink domain, a computer, such as an RPU, may bridge the two protocol domains by translating between CXL and UPLI. The computer includes a cache to store data retrieved from the UALink domain, allowing subsequent CXL requests for the same data to be processed from the cache without requiring additional cross-protocol translation or remote data fetching. This caching reduces latency during repeated accesses, decreases traffic on the UALink network, and improves overall system throughput. The computer can perform cache searches based on addresses and other identifiers included in the CXL request, and if the requested data is not present in the cache, it can translate opcodes, addresses, tags, and additional fields between CXL and UPLI messages.

[0256] In various implementations, the following techniques are used: a computer composed of caches receives a Compute Express Link (CXL) request from the first entity; the computer performs a cache lookup based on the CXL request; and in response to a cache miss, the computer translates the CXL request into an Ultra Accelerator Link Protocol Level Interface (UPLI) request; sends the UPLI request to another entity; receives a UPLI response from the second entity and stores the data in the cache; translates the UPLI response into a CXL response and sends the CXL response containing the data to the first entity in response to a cache hit: the computer sends the CXL response containing the data from the cache to the first entity, and does not send a corresponding UPLI request to the second entity. This technique may be performed by an RPU, semiconductor device, bridge, or other computing device located between the first and second entities. The cache may include an on-chip SRAM cache, an embedded DRAM cache, or a portion of memory allocated for caching. In the event of a cache miss, the computer translates the CXL opcode into the corresponding UPLI command, translates addresses between address spaces, or maps CXL tags to UPLI tags. When a cache hit occurs, a CXL response can be generated locally from the cached data, avoiding the delay and bandwidth overhead associated with cross-protocol round trips. The first entity includes CXL hosts, CXL devices, or CXL accelerators. The second entity includes accelerators, UALink switches, or other UPLI-enabled entities coupled via a UALink network.

[0257] In some implementations of this technique, a CXL request consists of a CXL.cache device-to-host (D2H) request containing a read opcode, and a CXL response consists of a CXL.cache host-to-device (H2D) response containing a global observation (GO) opcode and data. The first entity consists of a CXL device or CXL accelerator, and the second entity consists of an accelerator connected to a computer via a UALink network. The computer acts as a CXL host for the first entity, receiving D2H requests and responding with H2D responses and H2D data messages. Examples of read opcodes include RdOwn, RdShared, or RdAny. GO opcodes may indicate cache state granting, such as GO-S, GO-E, or GO-M.

[0258] In some implementations of this method, a CXL request consists of a CXL.mem Master-to-Subordinate (M2S) request containing MemRd*, and a CXL response consists of a CXL.mem Subordinate-to-Master Data Response (S2M DRS) consisting of MemData and data. The first entity then constitutes the CXL host. The computer acts as a CXL device subordinate to the CXL host, exposing a host-managed device memory (HDM) area backed by cache and UALink-accessible memory.

[0259] In some implementations of this technique, a CXL request consists of the first address in the first address space, and a UPLI request consists of the second address in the second address space translated from the first address. Here, the first address space consists of the host physical address (HPA) space, and the second address space consists of the network physical address (NPA) space. The computer can perform address translation using address range registers, translation tables, or algorithmic mappings. Furthermore, or alternatively, both addresses may reside within the same address space, such as the global address space or the partitioned global address space (PGAS).

[0260] In some implementations, this technique further involves the computer receiving a second CXL request from the first entity containing the write data; converting the second CXL request into a second UPLI request consisting of a write command and the write data; sending the second UPLI request to the second entity and responding to the second CXL request by either caching the write data in at least one of the caches or invalidating the data in the cache corresponding to the address of the second CXL request. The write command in the UPLI can include a Write or WriteFull command. Depending on the cache consistency policy, the computer may update the cache with the write data or invalidate the corresponding cache line to avoid stale data.

[0261] Some implementations of this method consist of instructions in a non-temporary, computer-readable medium that, when executed by a processor, cause the processor to execute the method. Some implementations of this technique involve one or more integrated circuits configured to execute this technique, and these one or more integrated circuits constitute one of the following: (i) a general-purpose processing unit configured or connected via a communication interface (configured to execute the method by software and / or firmware execution); (ii) firmware and / or hardware logic circuits integrated into electronic equipment, which utilize hardware acceleration or special processing capabilities not typically provided by general-purpose processing units; or (iii) the operation of having one or more chiplets within one or more integrated circuit packages.

[0262] The device includes a CXL port, an UALink port, a cache, and a computer linked to the port and cache, which can bridge the CXL and UALink protocol domains while caching data to reduce cross-protocol traffic. The CXL port can expose the CXL Type-2 or Type-3 device interface to the first entity, and the UALink port may connect to an UALink network consisting of one or more accelerators. The cache may reside inside the computer, or it may be located inside the device but outside the computer, or it may include a dedicated memory area accessible by the computer. The device may be implemented as a standalone device, as a chiplet in a multi-die processing unit, or as an IP block in an accelerator or switch.

[0263] Various implementations include the following devices: a Compute Express Link (CXL) port configured to communicate with the first entity according to CXL; an Ultra Accelerator Link (UALink) port configured to communicate with the second entity based on the Ultra Accelerator Link Protocol Level Interface (UPLI); a cache; and a computer connected to the CXL port, UALink port, and cache, where the computer is configured to: receive CXL requests from the first entity via the CXL port; perform cache lookups based on the CXL requests; and how to handle cache misses: convert the CXL request to a UPLI request, send the UPLI request to the second entity via the UALink port, receive a UPLI response containing data from the second entity, store the data in the cache, convert the UPLI response to a CXL response, and send the CXL response containing data to the first entity via the CXL port. Responding to cache hits: send the CXL response containing data from the cache to the first entity via the CXL port, and do not send a UPLI request corresponding to the CXL request to the second entity. The devices may be implemented as an RPU, bridge device, semiconductor device, or other hardware placed between the first and second entities. A CXL port may support one or more CXL subprotocols, such as CXL.cache, CXL.mem, and CXL.io. A UALink port may support UPLI commands such as Read, Write, WriteFull, and Atomic operations. The cache is indexed by address, and a computer can perform cache lookups by comparing the address of a CXL request with a tag stored in the cache. Upon a cache hit, the computer can generate a CXL response locally to avoid cross-protocol round-trip translation and network latency.

[0264] In some implementations, the CXL port is further configured to communicate according to CXL.cache, where a CXL request consists of a CXL.cache Device-to-Host (D2H) request containing a read opcode selected from RdOwn, RdShared, or RdAny, and a CXL response consists of a CXL.cache Host-to-Device (H2D) response consisting of a global observation (GO) opcode and a CXL.cache H2D data message. The first entity is the CXL device, the second entity is the accelerator, and the computer is the Resource Provisioning Unit (RPU). RdOwn can request data for exclusive ownership, RdShared for shared state data requests, and RdAny allows the host to determine the granted state. The GO opcode in the H2D response may be selected by the second entity based on the read opcode and cache line coherence state tracking.

[0265] In some implementations, the computer is further configured as follows: it receives a CXL.cache host-to-device (H2D) request via the CXL port that includes a snoop opcode targeting a cache line; it invalidates the cache line in response to the snoop opcode; and it sends a CXL.cache device-to-Host (D2H) response via the CXL port consisting of an opcode selected from RspIHitSE or RspIHitI. The snoop opcodes include SnpInv, SnpData, or SnpCur. Cache operations may include invalidating a cache line, downgrading the cache line from a higher state to a shared state, or preserving the current cache line state, depending on the snoop opcode and the state of the cache line when the snoop was received. RSP* includes RspIHitSE, RspIHitI, RspSHitSE, or RspVHitV, which can be selected based on the snoop opcode and the cache line state. In some cases, the Snoop opcode is SnpData, and a D2H data message may be sent along with the D2H response, such as when a computer transfers cached data to a second entity.

[0266] In some implementations of the device, the computer is further configured to evict data from the cache according to at least one eviction policy: one of the following: a Most Recently Used (LRU) replacement policy, a capacity-based eviction threshold, or a timer-based invalidation interval. Timer-based invalidation may be used when caching data transparently without host coherence tracking, for example, when using RdCurr. Eviction policies may be a combination of strategies, for example, using LRU replacement at a maximum occupancy threshold.

[0267] In some implementations of the device, the CXL port is configured to communicate further according to CXL.mem, where CXL requests consist of CXL.mem Master-to-Subordinate (M2S) requests containing MemRd*, and CXL responses consist of CXL.mem Subordinate-to-Master Data Response (S2M DRS) containing MemData. The first entity then consists of the CXL host. The device exposes the HDM region to the CXL host through CXL.mem, and the HDM region is backed by cache and UALink-accessible memory. The CXL host can access the HDM region using standard CXL.mem read operations.

[0268] In some implementations, the device is a multi-die processing unit, the computer is a resource provisioning unit (RPU) chiplet within the multi-die processing unit, and the CXL port is connected to the CXL root port of the multi-die processing unit via a coherent interconnect within the multi-die processing unit. The multi-die processing unit may contain chiplets coupled via the coherent interconnect, including compute chiplets, I / O chiplets, and RPU chiplets. The CXL root port may provide a CXL host interface for communication with external CXL devices or hosts.

[0269] When entities communicating according to UPLI, such as accelerators coupled via a UALink network, need to access data or memory resources in the CXL domain, such as memory coupled to a CXL host or CXL memory device, a computer like an RPU may bridge the two protocol domains by translating between UPLI and CXL. The computer includes a cache to store data retrieved from the CXL domain, allowing it to process subsequent UPLI requests for the same data from the cache without requiring additional cross-protocol translation or remote data fetching. This is particularly useful for workloads that repeatedly access the same data, such as artificial intelligence (AI) inference workloads, where accelerators access shared model weights, key-value (KV) cache entries, or attention parameters stored in CXL-connected memory.

[0270] When translating UPLI requests, different CXL opcodes can be used depending on the source of the request and the desired coherence behavior. For example, requests from the UALink network are translated with non-coherent opcodes such as RdCurr, and the CXL host does not recognize the cache copy. On the other hand, requests from local compute units within the same accelerator are translated with coherent opcodes such as RdShared, and the CXL host tracks the cache state and issues snoops to maintain coherence.

[0271] Various implementations employ the following techniques: A computer consisting of a cache receives an Ultra Accelerator Link Protocol Level Interface (UPLI) request from the first entity, consisting of a read command; the computer performs a cache lookup based on the UPLI request; responds to a cache miss: the computer converts the UPLI request into a Compute Express Link (CXL) request; sends the CXL request to another entity; receives a CXL response containing the data from the second entity and stores the data in the cache; converts the CXL response into a UPLI read response; and then sends the UPLI read response containing the data to the first entity. Responding to a cache hit: sends a UPLI read response containing the data from the cache to the first entity, and does not send a CXL request corresponding to the UPLI request to the second entity. This technique may be performed by an RPU, semiconductor device, bridge, or other computing device located between the first and second entities. In the event of a cache miss, the computer converts the UPLI read command into a CXL read opcode, performs address translation between address spaces such as NPA and HPA, and maps the UPLI ReqTag to a CXL tag or CQID. A UPLI read response may include an RdRsp carrying RdRspData and RdRspTag. When a cache hit occurs, a UPLI read response can be constructed locally from the cached data, embedding the RdRspTag from the original UPLI request and providing the cached data as RdRspData, thus avoiding cross-protocol conversion and CXL network latency. The first entity includes accelerators, UALink switches, or other UPLI-enabled entities. The second entity includes CXL hosts, CXL devices, or CXL memory devices.

[0272] In some implementations of this technique, a CXL request consists of a Device-to-Host (D2H) request for CXL.cache, including RdShared, and a CXL response consists of a CXL.cache host-to-device (H2D) response including GO-S and a CXL.cache H2D Data message containing the data. Furthermore, it includes a mechanism for the computer to respond by transitioning the cache line state in the cache from invalid (I) to shared (S) and storing the data in the cache. RdShared requests a cache line in the shared state, allowing a second entity to hold a cached copy of itself. The GO-S response gives the computer the shared state, making the second entity aware of the cached copy and allowing it to issue a snoop when consistency action is needed.

[0273] In some implementations of this technique, a CXL request consists of a CXL.cache device-to-host (D2H) request containing an RdCurr, and a CXL response consists of a CXL.cache host-to-device (H2D) data message containing the data. This means that storing data in the cache is transparent to the second entity, and the second entity does not maintain a consistent state of the data stored in the cache. The RdCurr retrieves the data without establishing a tracked coherence state in the second entity. Because the second entity is unaware of the cached copy, the computer may manage invalidation internally using timer-based expiration, capacity-based eviction, or software-driven invalidation.

[0274] In some implementations of this method, the UPLI request is received from the first entity via the UALink network, and the CXL request consists of the first CXL opcode selected based on the UPLI request received via the UALink network. Furthermore, it includes the computer translating requests received from the accelerator's local compute unit (CU) into a second CXL request containing a second CXL opcode different from the first CXL opcode. For example, a request from the UALink network might be translated to RdCurr for non-coherent access, and a request from the local CU might be translated to RdShared for coherent caching. The local CU may contain a streaming multiprocessor (SM), a compute engine, or other processing elements within the accelerator.

[0275] In some implementations, this technique further involves receiving a CXL.cache Host-to-Device (H2D) request from the second entity, which includes a SnpInv targeting the cached data; transitioning the cache line state in the cache from "Shared (S)" to "Invalid (I)"; and sending a CXL.cache Device-to-Host (D2H) response to the second entity, which includes an RspIHitSE. The SnpInv may be issued by the second entity if another agent requests exclusive ownership of the cache line. The RspIHitSE response indicates that the cache line has been found clean and invalidated, allowing the second entity to grant exclusive ownership to the requesting agent.

[0276] In some implementations of this method, a CXL request consists of a CXL.mem Master-to-Subordinate (M2S) request containing MemRd*, and a CXL response consists of a CXL.mem Subordinate-to-Master Data Response (S2M DRS) consisting of MemData and data. The second entity in this is the CXL memory device, which may include a CXL memory expander, a CXL memory pool, or a GFD. The computer may act as a CXL master to the CXL memory device, initiating an M2S request and receiving an S2M data response.

[0277] In some implementations of this method, the cached data consists of inference model data associated with an artificial intelligence (AI) model, which comprises either model weight parameters, key-value (KV) cache entries, activation data, or attention matrix coefficients. AI inference workloads may involve repeated access to the same model data by accelerators. Caching the inference model data on a computer reduces repeated cross-protocol fetches, lowers latency, reduces bandwidth consumption, and makes it available on both CXL and UALink networks.

[0278] In some implementations of this approach, the first entity consists of an accelerator made up of local memory, and the second entity provides access to CXL-connected memory. The cache provides an intermediate memory layer between the first entity's local memory and the CXL-connected memory, and the cache has lower latency accessing data than CXL-connected memory. Local memory may include HBM or other high-bandwidth memory directly connected to the accelerator. The cache provides fast access to frequently used data that does not fit in local memory and can mitigate the memory wall without incurring the full latency of CXL-connected memory access.

[0279] Some implementations of this method consist of instructions in a non-temporary, computer-readable medium that, when executed by a processor, cause the processor to execute the method. Some implementations of this technique involve one or more integrated circuits configured to execute this technique, and these one or more integrated circuits constitute one of the following: (i) a general-purpose processing unit configured or connected via a communication interface (configured to execute the method by software and / or firmware execution); (ii) firmware and / or hardware logic circuits integrated into electronic equipment, which utilize hardware acceleration or special processing capabilities not typically provided by general-purpose processing units; or (iii) the operation of having one or more chiplets within one or more integrated circuit packages.

[0280] The device may include UALink ports, CXL ports, a cache, and computers connected to the ports and cache, bridging the UALink and CXL protocol domains while caching data to reduce cross-protocol traffic. UALink ports may connect to a UALink network consisting of one or more accelerators, and CXL ports may connect to CXL hosts, CXL devices, or CXL memory devices. Computers may utilize different CXL subprotocols depending on the second entity; for example, using CXL.cache D2H requests when communicating with a CXL host, or CXL.mem M2S requests when communicating with a CXL memory device. The cache may support coherent caching based on host tracking status, transparent caching without host awareness, or both, depending on the CXL opcode used for data retrieval.

[0281] In various implementations, the following devices include: an Ultra Accelerator Link (UALink) port configured to communicate with a first entity compliant with the Ultra Accelerator Link Protocol Level Interface (UPLI); a Compute Express Link (CXL) port and cache configured to communicate with a second entity according to CXL; and a computer connected to the UALink port, CXL port, and cache, where the computer is configured to: receive UPLI requests containing read commands from the first entity via the UALink port; perform cache lookups based on the UPLI requests and handle cache misses: convert the UPLI request to a CXL request and send it to the second entity via the CXL port, receive a CXL response containing the data from the second entity, store the data in the cache, convert the CXL response to a UPLI read response, and send the UPLI read response containing that data to the first entity via the UALink port to respond to a cache hit: send the UPLI read response containing the data from the cache to the first entity via the UALink port, and do not send a CXL request to the second entity corresponding to the UPLI request. The device may be implemented as an RPU, bridge device, semiconductor device, or other hardware positioned between the first and second entities. A CXL port may support one or more CXL subprotocols such as CXL.cache, CXL.mem, and CXL.io. An UALink port may support UPLI commands such as Read, Write, WriteFull, and Atomic operations. When the cache is hit, the computer can construct the UPLI read response locally, avoiding cross-protocol translation and CXL network latency by embedding the RdRspTag from the original UPLI request and providing the cached data as RdRspData.UPLI's read command may include the Read command and the Read Class Vendor Defined Command.

[0282] In some implementations of the device, the CXL port is further configured to communicate according to CXL.cache, where a CXL request consists of a CXL.cache device-to-host (D2H) request with RdShared, and a CXL response consists of a CXL.cache host-to-device (H2D) response with GO-S and an H2D data message for CXL.cache. The first entity is the accelerator, the second entity is the CXL host, and the computer is a resource provisioning unit (RPU). The RPU acts as a CXL.cache device to the CXL host, issuing D2H requests and receiving H2D responses. Granting GO-S allows the CXL host to recognize the cached copy and issue snoops when coherency operations on the cached data are required.

[0283] In some implementations, the CXL port is further configured to communicate according to CXL.cache, where a CXL request consists of a CXL.cache device-to-host (D2H) request including an RdCurr, and a CXL response consists of an H2D data message from CXL.cache. Furthermore, storing data in the cache is transparent to a second entity, and the second entity does not maintain consistency of the data stored in the cache; here, the first entity is the accelerator and the second entity is the CXL host. The RdCurr returns the data without a GO response, and the CXL host is unaware of the cached copy. A computer can internally manage cache validity using elimination policies such as timer-based invalidation, capacity-based elimination, or software-driven invalidation.

[0284] In some implementations of the device, the computer is further configured as follows: Transition the cache line state from disabled (I) to shared (S) and respond if it receives a CXL.cache H2D response containing GO-S from a second entity. Transition the cache line state from shared (S) to disabled (I) and respond if it receives a CXL.cache host-to-device (H2D) request containing SnpInv from a second entity. Transition the cache line state from disabled (I) to exclusive (E), or from disabled (I) to modified (M) and respond if it receives a CXL.cache H2D response (GO-E or GO-M) from the latter entity. Cache state transitions may follow the semantics of the MESI protocol. GO-E or GO-M may be permitted if no other entity holds a cached copy of the cache line, or if the host determines that the exclusive or modified state is appropriate based on the access pattern.

[0285] In some implementations of the device, a UPLI request consists of the first address in the first address space, and a CXL request consists of the second address in the second address space translated from the first address. Here, the first address space consists of the network physical address (NPA) space, and the second address space consists of the host physical address (HPA) space. The computer can perform address translation using address range registers, translation tables, or algorithmic mappings. Furthermore, or alternatively, both addresses may reside within the same address space, for example, the global address space or a partitioned global address space.

[0286] In some implementations of the device, the CXL port is configured to communicate further according to CXL.mem, where CXL requests consist of CXL.mem Master-to-Subordinate (M2S) requests containing MemRd*, and CXL responses consist of CXL.mem Subordinate-to-Master Data Responses (S2M DRS) containing MemData. The second entity within this is the CXL memory device, which may include a CXL memory expander, a CXL memory pool, or a GFD. The computer may act as the CXL master to the CXL memory device, caching the retrieved data to reduce repeated access across the CXL link.

[0287] In some implementations, the device is a multi-die processing unit, the computer is a resource provisioning unit (RPU) chiplet within the multi-die processing unit, and the CXL port is connected to the CXL root port of the multi-die processing unit via a coherent interconnect within the multi-die processing unit. The multi-die processing unit may contain chiplets coupled via the coherent interconnect, including compute chiplets, I / O chiplets, and RPU chiplets. The CXL root port may provide a CXL host interface for communication with external CXL devices or hosts.

[0288] Figure 19A shows an example system consisting of a first entity (Entity.1 / Accelerator.1 / GPU.1), a second entity (Entity.2 / Accelerator.2 / GPU.2) consisting of an UALink switch (ULS) and a cached RPU, and a third entity (Entity.3 / Host) connected to memory. Entity.1 includes a GPU, accelerator, compute element, host, CPU, multi-die processing unit, UALink switch, originator, or consumer. Entity.3 includes a host, CPU, GPU, accelerator, CXL switch, compute element, multi-die processing unit, memory pool, provider, etc. Entity.1 is connected to the UALink switch via an UALink connection. The UALink switch is connected to Entity.2 via an UALink connection. Entity.2 consists of an RPU and is connected to Entity.3 via a CXL.cache connection. Entity.3 is connected to memory. Figure 19B shows an example of a TFD that distinguishes requests from different sources in an RPU (RPU and cache) including a cache, and converts them into CXL.cache D2H requests with different opcodes based on the source of the request. The TFD shows two transaction sequences: a request sequence from the UALink network converted to RdCurr, and a request sequence from a local compute unit (CU) converted to RdShared.

[0289] In the UALink network request sequence (represented by circles 1 to 5), Entity.1 sends a UPLI request (ReqLen) to the RPU (circle 1) consisting of a read command (ReqCmd(Read)), source physical accelerator identifier (ReqSrcPhysAccID(a.1)), destination physical accelerator identifier (ReqDstPhysAccID(b.1)), request address in the network physical address space (ReqAddr(AS.1.1 / NPA)), request tag (ReqTag(c.1.1)), and request length (ReqLen(d.1.1)). The RPU performs a cache lookup and determines if a cache miss occurs. In response to a cache miss, the RPU converts the UPLI request into a D2H request in CXL.cache, consisting of RdCurr, CQID(q.2.1), and address in the host physical address space (Address(AS.2.1 / HPA)) (circle 2). The RPU can perform address translation from the NPA space (AS.1.1 / NPA) to the HPA space (AS.2.1 / HPA). In some cases, the RPU may also perform intermediate address translation from the NPA to the SPA (System Physical Address) and then to the HPA. Entity.3 responds with a CXL.cache H2D Data message consisting of the CQID (q.2.1) and the requested data (Data(*Data*)) (Circle 3). Because RdCurr is used, Entity.3 does not send a GO response, and the data is not cached in a host-tracked coherency state. The annotation "Not Cached (RdCurr)" and the I-to-I transition (Circle 4) indicate that Entity.3 may or may not cache the data transparently without maintaining the coherency state of a cached copy. The RPU converts the H2D data in CXL.cache into a UPLI read response / data (RdRsp), which consists of RdRspSrcPhysAccID(b.1), RdRspDstPhysAccID(a.1), RdRspTag(c.1.1), and RdRspData(*Data*). This UPLI RdRsp is sent to Entity.1 (Circle 5).

[0290] In the local CU request sequence (indicated by circles 6 through 10), a compute unit (CU) within the accelerator sends a read request to the RPU (circle 6) containing an address in the local address space (e.g., the guest virtual address (GVA) space (Addr(AS.3.1 / GVA))). The RPU performs a cache lookup and determines if there is a cache miss (circle between 6 and 7). In response to a cache miss, the RPU returns the CU request to CXL.cache It is translated into a D2H request and consists of RdShared, CQID(q.4.1), and the address in the host physical address space (Address(AS.4.1 / HPA)) (Circle 7). The RPU can perform address translation from the GVA space (AS.3.1 / GVA) to the HPA space (AS.4.1 / HPA). Entity 3 responds with an H2D response in CXL.cache. This consists of GO-S, a response data value indicating the shared state (RspData(S)), and CQID(q.4.1) (Circle 8), followed by a CXL.cache H2D Data message (CQID(q.4.1) and the requested data (Data(*Data*)) (Circle 9). The RPU stores the data in the cache and transitions the cache line state from invalid (I) to shared (S) (represented by the transition from I to S between Circles 7 and 9). The RPU provides the requested data to the CU (Circle 10).

[0291] The distinction between the RdCurr opcode for UALink network requests and the RdShared opcode for local CU requests reflects the different coherency requirements of each source. Requests from the UALink network may be I / O coherent and may not benefit from the host tracking cache, so the RPU may translate them to RdCurr to retrieve data without establishing coherency state tracked by Entity.3. Requests from local CUs are related to the accelerator's cache hierarchy and benefit from the coherent cache, so the RPU translates them to RdShared, establishing shared state tracked by Entity.3 and allowing Entity.3 to issue snoops during coherency actions.

[0292] In environments wher...

Claims

1. The methods for converting Ultra Accelerator Link (UALink) protocol-level interface (UPLI) requests to Compute Express Link (CXL) requests include: According to UPLI, it means communicating with the first entity; According to CXL.mem, it means communicating with another entity; The first entity sends an UPLI request consisting of a read command and the first physical address. The UPLI request is translated into a request from the master to the sublesson of CXL.mem, including MemRd* and the second physical address (CXL.mem M2S Req MemRd*). Send CXL. mem M2S Req MemRd* to the second entity.

2. The method of claim 1 further includes a ReqSrcPhysAccID field, a ReqDstPhysAccID field, a ReqLen field, a ReqTag field, a ReqAddr field containing the first physical address, and a ReqCmd field containing the read command. Furthermore, it also includes converting the ReqTag to a tag associated with CXL. mem M2S Req.

3. The method of claim 1 further receives a CXL.mem subordinate-to-master data response (S2M DRS) consisting of MemData, tags, and data from the following entity; converts the CXL.mem S2M DRS into an UPLI read response (RdRsp), consisting of RdRspSrcPhysAccID field, RdRspDstPhysAccID field, RdRspTag field, and RdRspData; and sends the UPLI RdRsp to the first entity.

4. The method of claim 3 further includes a Poison field in CXL.mem S2M DRS and an RdRsp Data Error field in UPLI RdRsp. Furthermore, it also includes converting the Poison field of CXL.mem S2M DRS to the RdRsp Data Error field of UPLI RdRsp.

5. The method of claim 1 is such that the read command is Read Class Vendor Defined Command, the first entity is an accelerator or UALink switch, the second entity is a CXL device, and the second physical address is the host physical address (HPA) used by the second entity.

6. The method of claim 1, wherein the UPLI request indicates an I / O coherent read, and CXL. mem M2S Req MemRd* further includes a SnpType field containing SnpCur, a MetaField field containing Meta0-State (MS0), and a MetaValue field containing Invalid (I).

7. The method of claim 1 further includes sending a CXL. mem M2S request (MemSpecRd.) to the latter entity.

8. In the method of claim 1, the first physical address is associated with the first address space, the second physical address is associated with the second address space which is different from the first address space, and the conversion further converts the first physical address to the second physical address.

9. It consists of instructions in a non-temporary computer-readable medium that, when executed by a processor, perform the method of claim 1.

10. One or more integrated circuits configured to perform the method of claim 1, wherein one or more integrated circuits include any of the following: (i) a general-purpose processing unit including a communication interface and configured to perform the method by software and / or firmware execution; (ii) firmware and / or hardware logic circuits integrated into an electronic device, which utilize hardware acceleration or special processing capabilities not typically provided by the general-purpose processing unit; or (iii) the operation of mounting one or more chiplets in one or more integrated circuit packages.

11. An active cable comprising first and second pluggable modules coupled by a physical medium; wherein the active cable further includes hardware circuitry integrated into the active cable and configured to perform the method of claim 1.

12. A device configured to operate as a switch, comprising a switching circuit, and configured to perform the method of claim 1.

13. Systems including the following: The processing unit consists of an Ultra Accelerator Link (UALink) port, a memory controller connected to the first memory, and a Compute Express Link (CXL) port connected to the second memory. Here, the UALLink port is configured to communicate with entities according to the UALLink-based protocol. The processing unit is configured to provide the first memory path from the UALink port via the memory controller, and the second memory path from the UALink port via the CXL port to the second memory. The processing unit further configures a resource provisioning unit (RPU) which is set up to receive the initial UALink protocol-level interface (UPLI) request from the entity, forward the initial translation request to the first memory via the first memory path, receive a second UPLI request from the entity, and forward the second translation request to the second memory via the second memory path.

14. In the system of claim 13, the processing unit further constitutes a coherent interconnect, and the first memory path and the second memory path pass through a portion of the coherent interconnect.

15. In the system of claim 14, the CXL port is connected to the CXL / PCIe root port (RP) and the coherent interconnect.

16. The system of claim 14 further includes a processing unit comprising a processing core, a cache / home agent (CHA), a snoop filter (SF), and a last-level cache (LLC) slice, and connected to a coherent interconnect. Furthermore, it includes either a PCIe root port connected to an I / O device or an intersocket link (ISoL) port connected to a second processing unit.

17. The system of claim 13, namely the UALink port, CXL port, and memory controller, is located within the same integrated circuit (IC) package; furthermore, the RPU translates the physical addresses associated with the UALink-based protocol to the physical addresses associated with the processing unit, and is configured so that entities can access the first and second memories.

18. In the system of claim 13, the second translation request consists of a CXL.mem Master-to-Subordinate (M2S) request (MemRd*), and the CXL port is configured to send the CXL.mem M2S request to a second memory. The CXL port is further configured to receive a CXL.mem lower-to-master data response (S2M DRS) containing the memory data and data from the second memory, and the RPU is configured to translate the CXL.mem S2M DRS into an UPLI read response (RdRsp) containing the data, and to send the UPLI RdRsp to the entity.

Images