Telemetry and load balancing in CXL systems

Abstract

A system may include a host configured to provide requests to a plurality of computing resources and receive responses from the plurality of computing resources. In an example, the computing resources may be distributed over respective accelerator devices, which may be configured to communicate with the host using various protocols, such as using a computing high speed link (CXL). A first accelerator device may include a telemetry manager that may receive a queue utilization signal indicating an amount of transaction request messages or response messages processed by the first accelerator device. The first accelerator device may determine a device load metric for the first accelerator device based on the queue utilization signal, and may provide a control signal to the host device with information about the device load metric. The host device may select the first accelerator device or a different device based on the control signal.

Description

Technical Field Background Art

[0001] Memory devices used in computers or other electronic devices can be classified as volatile and non-volatile memory. Volatile memory requires power to maintain its data, and includes random access memory (RAM), dynamic random access memory (DRAM), or synchronous dynamic random access memory (SDRAM), etc. Non-volatile memory can retain stored data when not powered, and includes flash memory, read-only memory (ROM), electrically erasable programmable ROM (EEPROM), static RAM (SRAM), erasable programmable ROM (EPROM), resistance variable memory, phase change memory, storage class memory, resistive random access memory (RRAM), and magnetoresistive random access memory (MRAM), etc. Persistent memory is an architectural property of the system, where the data stored in the media is available after a system reset or power cycle. In some examples, non-volatile memory media can be used to build a system with a persistent memory model.

[0002] The memory device may be coupled to a host (e.g., a host computing device) to store data, commands, and / or instructions for use by the host while the computer or electronic system is running. For example, data, commands, and / or instructions may be transferred between the host and the memory device during operation of a computing or other electronic system.

[0003] Various protocols or standards may be applied to facilitate communication between a host and one or more other devices such as memory buffers, accelerators, or other input / output devices. In an example, an out-of-order protocol such as Compute Express Link (CXL) may be used to provide high bandwidth and low latency connectivity. Summary of the invention BRIEF DESCRIPTION OF THE DRAWINGS

[0004] To easily identify the discussion of any particular element or act, the most significant digit(s) in a reference number refers to the figure number in which the element is first introduced.

[0005] Figure 1 A block diagram of an example computing system including a host and a memory device is generally illustrated.

[0006] Figure 2 An example of a compute express link (CXL) system is generally described.

[0007] Figure 3 An example of a CXL system implementing a virtual hierarchy for managing transactions is generally described.

[0008] Figure 4A , Figure 4B , Figure 5A , Figure 5B and Figure 5C Respective examples of a CXL host device coupled to an accelerator device including a memory controller are generally described.

[0009] Figure 6 An example of an accelerometer device including a telemetry manager and a thermal manager is generally described.

[0010] Fig. 7A , Figure 7B and Figure 7C Respective examples of a CXL host device coupled to an accelerator device including a memory device are generally described.

[0011] Figure 8 An example of a method for determining a device loading metric is generally described.

[0012] Fig. 9 An example of a method for using a device loading metric is generally described.

[0013] Fig.10 A block diagram illustrating an example machine that may be utilized, in, or by which any one or more of the techniques discussed herein may be implemented. DETAILED DESCRIPTION

[0014] Compute Express Link (CXL) is an open standard interconnect configured for high-bandwidth, low-latency connectivity between host devices and other devices such as accelerators, memory buffers, and other I / O devices. CXL is designed to facilitate high-performance computing workloads by supporting heterogeneous processing and memory systems. CXL implements coherency and memory semantics on top of PCI Express (PCIe)-based I / O semantics for optimized performance.

[0015] In some instances, CXL is used in applications such as artificial intelligence, machine learning, analytics, cloud infrastructure, edge computing devices, communication systems, etc. Data processing in such applications may use various scalar, vector, matrix, and spatial architectures that may be deployed in CPUs, GPUs, FPGAs, smart NICs, or other accelerators that may be coupled using CXL links.

[0016] CXL supports dynamic multiplexing using a set of protocols that include input / output (CXL.io over PCIe), cache (CXL.cache), and memory (CXL.memory or CXL.mem) semantics. In an example, CXL can be used to maintain a unified, consistent memory space between a CPU (e.g., a host device or host processor) and any memory on an attached CXL device. This configuration allows the CPU and CXL devices to share resources and operate on the same memory region for higher performance, reduced data movement, and reduced software stack complexity. In an example, the CPU is primarily responsible for maintaining or managing consistency in a CXL environment. Therefore, CXL can be utilized to help reduce device cost and complexity, as well as the overhead traditionally associated with consistency across I / O links.

[0017] CXL runs on top of the PCIe PHY and provides full interoperability with PCIe. In an example, a CXL device starts link training at the PCIe Gen 1 data rate and, if its link partner supports CXL, negotiates CXL as its operating protocol (e.g., using the alternative protocol negotiation mechanism defined in the PCIe 5.0 specification). As a result, devices and platforms can more easily adopt CXL by leveraging the PCIe infrastructure, and without having to design and verify PHYs, channels, channel extension devices, or other upper layers of PCIe.

[0018] In an example, CXL supports single-level switching to enable fan-out to multiple devices. This enables multiple devices in a platform to be migrated to CXL while maintaining backward compatibility and the low latency characteristics of CXL. In an example, CXL can provide a standardized computing structure that supports the aggregation of multiple logical devices (MLDs) and single logical devices, such as using CXL switches connected to several host devices or nodes (e.g., root ports). This feature enables servers to aggregate resources that can be assigned based on workloads, such as accelerators and / or memory. For example, CXL can help facilitate resource allocation or dedication and release. In an example, CXL can help allocate and deallocate memory to various host devices as needed. This flexibility helps designers avoid over-provisioning while ensuring optimal performance.

[0019] Some of the compute-intensive applications and operations mentioned herein may require or use large data sets. Memory devices storing such data sets may be configured for low latency and high bandwidth and persistence. One problem with load-store interconnect architectures involves ensuring persistence. CXL can help solve this problem using software architectural flows and standard memory management interfaces, for example enabling persistent memory to move from a controller-based approach to direct memory management.

[0020] The inventors have recognized that a problem to be solved includes resource provisioning in a system that uses CXL to coordinate operations between multiple devices, such as using one or more hosts or one or more accelerators, which may, for example, include corresponding memory devices. In an example, the problem may include enhancing or optimizing telemetry in a CXL system. Telemetry generally includes automated or semi-automated transmission of information between components in a system to help improve performance, safety, or other system metrics. Telemetry information can be used to govern and manage resources connected in a CXL system, and may include indications of under-provisioned (underutilized) and over-provisioned (overutilized) resources. In some examples, telemetry information may include accelerator performance, utilization, temperature, or other information that can be used to characterize or quantify the behavior of individual components or accelerators in the system.

[0021] In an example, telemetry information or quality of service (QoS) telemetry information may include information about each of the multiple threads. For example, the telemetry information may include information about which of the multiple threads is accessing a particular resource (e.g., memory) or is using system bandwidth or occupying a particular computing resource in the system.

[0022] The inventors have recognized that the problem may include providing telemetry services in a CXL-based system. Such a system may include one or more media controllers (e.g., memory controllers), cache subsystems, or other accelerators. In an example, the problem may include providing queue-specific telemetry information in a CXL-based system that may include or use virtualization, such as using a multiple logical device (MLD) or other virtual CXL switch or switch module. The problem may include, for example, providing telemetry for memory transactions in a virtualized or partially virtualized CXL system.

[0023] The inventors have recognized that a solution to the telemetry problem may include or use resource reports from each of a plurality of devices or queues in a CXL system. The solution may further include or use thermal status information to help provide information about the loading of a particular resource. The solution may optionally include providing resource usage or thermal status information for a request path, a response path, or a request and response path through the CXL system. In an example, the solution includes or uses a telemetry manager configured to track utilization or usage, thermal status, and read-write response ratios of queues in a particular accelerator coupled to a computing structure, which may include, for example, a CXL-based system. In an example, each accelerator (or each functional portion thereof) may be classified according to utilization, for example, based on programmable thresholds indicating different resource utilization levels. The levels may include, for example, light, optimal, moderate, and heavy device utilization or loading levels. The levels or thresholds may be based, for example, on internal device queue utilization, read-write traffic ratios, or thermal characteristics, etc. In an example, accelerator-specific resources may be monitored or measured at various locations in a particular accelerator device, for example, at the input of a CXL controller or at the output of a media controller (e.g., a memory controller). In an example, the solution may include or use a respective telemetry manager for each categorized resource scope.

[0024] Figure 1 A block diagram generally illustrates an example of a computing system 100 including a host device 102 and a memory system 104. The host device 102 includes a central processing unit (CPU) or processor 110 and a host memory 108. In an example, the host device 102 may include a host system such as a personal computer, a desktop computer, a digital camera, a smart phone, a memory card reader, and / or an Internet of Things enabled device, as well as various other types of hosts, and may include a memory access device, such as the processor 110. The processor 110 may include one or more processor cores, a system of parallel processors, or other CPU arrangements.

[0025] The memory system 104 includes a controller 112, a buffer 114, a cache 116, and a first memory device 118. The first memory device 118 may include, for example, one or more memory modules (e.g., a single in-line memory module, a dual in-line memory module, etc.). The first memory device 118 may include volatile memory and / or non-volatile memory, and may include a multi-chip device that includes one or more different memory types or modules. In an example, the computing system 100 includes a second memory device 120 that interfaces with the memory system 104 and the host device 102.

[0026] Host device 102 may include a system backplane and may include multiple processing resources (e.g., one or more processors, microprocessors, or some other type of control circuitry). Computing system 100 may optionally include separate integrated circuits for host device 102, memory system 104, controller 112, buffer 114, cache 116, first memory device 118, second memory device 120, any one or more of which may include respective chiplets that may be connected and used together. In an example, computing system 100 includes a server system and / or a high performance computing (HPC) system and / or a portion thereof. Although Figure 1 The examples shown in illustrate a system having a Von Neumann architecture, but embodiments of the present disclosure may be implemented in a non-Von Neumann architecture, which may not include one or more components (e.g., CPU, ALU, etc.) typically associated with a Von Neumann architecture.

[0027] In an example, the first memory device 118 may provide main memory for the computing system 100, or the first memory device 118 may include auxiliary memory or storage for use by the computing system 100. In an example, the first memory device 118 or the second memory device 120 includes one or more arrays of memory cells, such as volatile and / or non-volatile memory cells. For example, the array may be a flash array with a NAND architecture. Embodiments are not limited to a particular type of memory device. For example, the memory device may include RAM, ROM, DRAM, SDRAM, PCRAM, RRAM, and flash memory, among others.

[0028] In embodiments where the first memory device 118 includes persistent or non-volatile memory, the first memory device 118 may include a flash memory device, such as a NAND or NOR flash memory device. The first memory device 118 may include other non-volatile memory devices, such as non-volatile random access memory devices (e.g., NVRAM, ReRAM, FeRAM, MRAM, PCM), memory devices such as ferroelectric RAM devices including ferroelectric capacitors that may exhibit hysteresis characteristics, 3D cross point (3D XP) memory devices, etc., or combinations thereof.

[0029] In an example, the controller 112 includes a media controller, such as a non-volatile memory express (NVMe) controller. The controller 112 may be configured to perform operations such as copying, writing, reading, error correction, etc. on the first memory device 118. In an example, the controller 112 may include circuit systems and / or instructions for specific purposes to perform various operations. That is, in some embodiments, the controller 112 may include circuit systems and / or may be configured to execute instructions to control the movement of data and / or addresses associated with the data, such as among the buffer 114, the cache 116, and / or the first memory device 118 or the second memory device 120.

[0030] In an example, at least one of the processor 110 and the controller 112 includes a command manager (CM) for the memory system 104. The CM may, for example, receive a read command for a specific logical row address in the first memory device 118 or the second memory device 120 from the host device 102. In some examples, the CM may determine that the logical row address is associated with the first row based at least in part on a pointer stored in a register of the controller 112. In an example, the CM may receive a write command for the logical row address from the host device 102, and the write command may be associated with the second data. In some examples, the CM may be configured to issue an access command associated with the first memory device 118 or the second memory device 120 to the non-volatile memory and issue the access command between issuing the read command and the write command. In some examples, the CM may issue an access command associated with the first memory device 118 or the second memory device 120 to the non-volatile memory and issue the access command between issuing the read command and the write command.

[0031] In an example, buffer 114 includes data buffer circuitry that includes an area of ​​physical memory for temporarily storing data, such as when the data is moved from one location to another. Buffer 114 may include a first-in, first-out (FIFO) buffer, where the oldest (e.g., first-in) data is processed first. In some embodiments, buffer 114 includes a hardware shift register, a circular buffer, or a list.

[0032] In an example, the cache 116 includes an area of ​​physical memory for temporarily storing specific data that is likely to be used again. The cache 116 may include a pool of data entries. In some examples, the cache 116 may be configured to operate according to a write-back policy, in which data is written to the cache without being simultaneously written to the first memory device 118. Therefore, in some embodiments, data written to the cache 116 may not have a corresponding data entry in the first memory device 118.

[0033] In an example, the controller 112 may receive write requests related to the cache 116 (e.g., from the host device 102) and cause data associated with each of the write requests to be written to the cache 116. In some examples, the controller 112 may receive write requests at a rate of thirty-two (32) gigatransfers (GT) / second, e.g., according to or using the CXL protocol. The controller 112 may similarly receive read requests and cause data stored in, for example, the first memory device 118 or the second memory device 120 to be retrieved and written to, for example, the host device 102 via the interface 106.

[0034] In an example, the interface 106 may include any type of communication path, bus, etc. that allows information to be transferred between the host device 102 and the memory system 104. Non-limiting examples of interfaces may include a peripheral component interconnect (PCI) interface, a peripheral component interconnect express (PCIe) interface, a serial advanced technology attachment (SATA) interface, and / or a micro serial advanced technology attachment (mSATA) interface, etc. In an example, the interface 106 includes a PCIe5.0 interface that complies with the computing express link (CXL) protocol standard. Therefore, in some embodiments, the interface 106 supports a transfer speed of at least 32GT / s.

[0035] As similarly described elsewhere herein, CXL is a high-speed central processing unit (CPU) to device or CPU to memory interconnect designed to enhance computing performance. CXL technology maintains memory coherency between the CPU memory space (e.g., host memory 108) and the memory on the attached device or accelerator (e.g., first memory device 118 or second memory device 120), which allows resource sharing for higher performance, reduced software stack complexity, and lower overall system cost. As accelerators are increasingly used to supplement CPUs to support emerging data-rich and compute-intensive applications such as artificial intelligence and machine learning, CXL is designed as an industry open standard interface for high-speed communication.

[0036] Figure 2 An example of a CXL system 200 is generally described in which a host device 202 and a CXL device 204 are connected using a CXL link 206. In the example, according to Figure 1 In an example of computing system 100 in FIG. 1 , host device 202 includes or corresponds to host device 102, and CXL device 204 includes or corresponds to memory system 104. A memory system command manager (CM) may include a portion of host device 202 or CXL device 204. In an example, CXL link 206 (e.g., corresponding to a CXL link from Figure 1The interface 106 of an instance of the CXL device 204 may support communication using a multiplexed protocol for caching (e.g., CXL.cache), memory access (e.g., CXL.mem or CXL.memory), and data input / output transactions (e.g., CXL.io). CXL.io may include a PCIe-based protocol for functions such as device discovery, configuration, initialization, I / O virtualization, and direct memory access (DMA) using non-uniform load-store, producer-consumer semantics. CXL.cache may enable a device to cache data from a host memory (e.g., from the host memory 212) using a request and response protocol. CXL.memory may enable a host device 202 to use memory attached to the CXL device 204, for example, in or using a virtualized memory space. CXL-based memory devices may include or use volatile or non-volatile memory, which may be characterized by different speeds or latencies, for example. In an example, a CXL-based memory device may include a CXL-based memory controller configured to manage transactions with volatile or non-volatile memory.

[0037] In an example, CxL.memory transactions may be memory load and store operations that are run downstream or external to the host device 202. CXL memory devices may have different levels of complexity. For example, a simple CXL memory system may include a CXL device that includes or is coupled to a single media controller, such as a memory controller (MEMC). A moderate CXL memory system may include a CXL device that includes or is coupled to multiple media controllers. A complex CXL memory system may include a CXL device that includes or is coupled to a cache controller (and its accompanying caches) and is coupled to one or more media or memory controllers.

[0038] exist Figure 2 In an example of , host device 202 includes a host processor 214 (e.g., including one or more CPUs or cores) and an IO device 226. Host device 202 may include or may be coupled to host memory 212. Host device 202 may include various circuitry or logic configured to facilitate CXL-based communications and transactions with CXL device 204. For example, host device 202 may include coherence and memory logic 218 configured to implement transactions according to CXL.cache and CXL.memory semantics, and host device 202 may include PCIe logic 220 configured to implement transactions according to CXL.io semantics. In an example, host device 202 may be configured to manage the coherence of data cached at CXL device 204 using, for example, its coherence and memory logic 218.

[0039] Host device 202 may further include a host multiplexer 216 configured to modulate communications over CXL link 206 (e.g., using a PCIe PHY layer). Multiplexing of protocols ensures that latency-sensitive protocols (e.g., CXL.cache and CXL.memory) have the same or similar latency as the native inter-processor link. In an example, CXL places an upper limit on the response time of latency-sensitive protocols to help ensure that device performance is not adversely affected by latency variations between different devices implementing coherency and memory semantics.

[0040] In an example, a symmetric cache coherence protocol may be difficult to implement between host processors because different architectures may use different solutions, which in turn may compromise backward compatibility. CXL may address this issue by merging coherence functions at host device 202, such as using coherence and memory logic 218.

[0041] The CXL device 204 may include an accelerator device including various accelerator logic 222. In an example, the CXL device 204 may include a device memory 228a, or may be coupled to a CXL device memory 228b. The host device 204 may include various circuitry or logic configured to facilitate CXL-based communications and transactions with the host device 202 using the CXL link 206. For example, the accelerator logic 222 may be configured to implement transactions according to CXL.cache, CXL.memory, and CXL.io semantics. The CXL device 204 may include a CXL device multiplexer 224 configured to control communications over the CXL link 206.

[0042] In an example, one or more of the coherence and memory logic 218 and the accelerator logic 222 include a unified assistance engine (UAE) or a compute structure having various functional units, such as a command manager (CM), a thread engine (TE), a streaming engine (SE), a data manager or data mover (DM), or other units. The compute structure may be reconfigurable and may include separate synchronous and asynchronous flows.

[0043] The accelerator logic 222 or portions thereof may be configured to operate in the application space of the CXL system 200 and, in some instances, may launch its own threads or sub-threads that may operate in parallel and may optionally use resources or units on other CXL devices 204. Queue and transaction control through the system may be coordinated by the CM, TE, SE, or DM components of the UAE. In an example, each queue or thread may be mapped to a different loop iteration, thereby supporting multi-dimensional loops. With the ability to launch such nested loops, among other capabilities, the system may achieve significant time savings and latency improvements for compute-intensive operations.

[0044] In an example, command fences may be used to help maintain order throughout such operations, which may be performed locally or throughout the computational space of accelerator logic 222. In some examples, CM may be used to route commands to a specific command execution unit (e.g., accelerator logic 222 including a specific instance of CXL device 204) using an out-of-order interconnect that provides corresponding transaction identifiers (TIDs) for command and response message pairs.

[0045] In an example, the CM may coordinate synchronization flows, such as using an asynchronous structure of a reconfigurable computing structure to communicate with other synchronization flows and / or other components of the reconfigurable computing structure using asynchronous messages. For example, the CM may receive an asynchronous message from a dispatch interface and / or from another flow controller indicating a new thread at or using a synchronization flow. The scheduling interface may interface between the reconfigurable computing structure and other system components. In some examples, a synchronization flow may send an asynchronous message to a dispatch interface to indicate completion of a thread.

[0046] The synchronization messages may be used by the synchronization streams, for example, to access memory. For example, the reconfigurable computing structure may include one or more memory interfaces. The memory interface is a hardware component that can be used by the synchronization stream or its components to access external memory that is not part of the synchronization stream but can be accessed by the host device 202 or the CXL device 204. The threads executed using the synchronization streams may include sending read and / or write requests to the memory interface. Because reads and writes are asynchronous, the thread that initiates a read or write request to the memory interface may not receive the results of the request. Alternatively, the results of the read or write request may be provided to different threads executing at different synchronization streams. Delay and output registers in one or more of the CXL devices 204 may help coordinate and maximize the efficiency of the first stream, for example, by accurately timing the participation of specific computing resources of one device and the arrival of data related to the first stream. The registers may help enable specific computing resources of the same resource to be used for streams other than the first stream, such as when the first stream is stopped or waiting for other data or operations to complete. Such other data or operations may depend on one or more other resources of the structure.

[0047] In an example, the CXL devices 204 include memory devices that are configured to report their load or occupancy status to the host device 202 or to other devices or managers in the CXL system 200 using quality of service (QoS) telemetry. In an example, each CXL device 204 reports its load status to its respective host in each CXL.mem response. The host device 202 may receive load status information from the CXL device 204 and from one or more other CXL devices 204 in the CXL system 200, and use the information together to help control traffic in the CXL system 200 and avoid underutilizing (e.g., undercompensating or underprovisioning) or overutilizing (e.g., overcompensating or overprovisioning) resources in the system. The host device 202 may be configured to dynamically adjust its request processing in response to the load status information from one or more CXL devices 204 to help optimize system performance.

[0048] In an example, each CXL device 204 may be configured to determine its load state (sometimes referred to as device load or DevLoad) based on internal resource utilization metrics or internal queuing (e.g., within one or more device subsystems). A metric for analyzing utilization may include a continuous spectrum of quantitative DevLoad characteristics. In an example, another metric for analyzing utilization may include or use categories or groups of device usage, such as may be based on different specified threshold amounts or quantities of particular DevLoad characteristics. For example, a minimum or "light" DevLoad may correspond to a minimum queuing delay within a CXL device 204, and thus the CXL device 204 may indicate underutilization and availability to accept additional transactions or requests from the host device 202. An "optimal" DevLoad may correspond to a moderate or medium queuing delay within a CXL device 204, and thus the CXL device 204 may indicate to the host device 202 that it is optimally utilized. A "medium" DevLoad may correspond to a significant queuing delay inside the CXL device 204, and thus the CXL device 204 may indicate to the host device 202 that resources on the CXL device 204 are overprovisioned and additional requests will be queued or delayed. A "heavy" DevLoad may correspond to the highest queuing delay inside the CXL device 204, and thus the CXL device 204 may indicate to the host device 202 that resources on the CXL device 204 are severely overprovisioned and additional requests will be queued and significantly delayed. Other categories or groups may be similarly defined with various intermediate or overlapping levels of device usage or DevLoad granularity.

[0049] Some telemetry systems may be configured to track, for example, memory resource queues within a particular CXL device 204 to determine the DevLoad characteristics of a particular device. The inventors have recognized that in some systems, such as may include or use virtualized memory spaces in a CXL system, various device internal queues and / or read-write traffic ratios may be monitored and used to help optimize resource provisioning and, therefore, improve system performance.

[0050] Figure 3 An example of a portion of a CXL system is generally described that may include or use a virtual hierarchy to manage transactions, such as memory transactions with CXL memory devices. The example may include or use real-time telemetry to help facilitate allocation of new or ongoing queues. Figure 3 Examples of include first virtual hierarchy 304 and second virtual hierarchy 306. First virtual hierarchy 304, second virtual hierarchy 306, or one or more modules or components thereof may be implemented using host device 202, CXL device 204, or multiple instances of host device 202 or CXL device 204.

[0051] exist Figure 3 In the example of FIG. 3 , the first virtual hierarchy 304 includes a first host device 308, and the second virtual hierarchy 306 includes a second host device 310. A CXL switch 302 may be provided to expose a plurality of CXL resources to different hosts in the system. In other words, the CXL switch 302 may be configured to couple each of the first host device 308 and the second host device 310 to the same or different resources, for example using respective virtual CXL switches (VCSs), such as first VCS 320 and second VCS 322, respectively. The CXL switch 302 may be statically configured to couple each host device to respective different resources, or the CXL switch 302 may be dynamically configured to different resources, for example depending on the need of a particular one of the host devices to execute its respective queue or thread. Thus, the CXL switch 302 enables virtual hierarchies and resource sharing between different hosts.

[0052] In an example, a fabric manager (FM) may be provided to assign or coordinate the connections of CXL switches 302, and the FM may be configured to initiate, dismantle, or reconfigure the virtual hierarchy of the CXL system. The FM may include a baseboard management controller (BMC), an external controller, a centralized controller, or other controller.

[0053] exist Figure 3In the example of FIG. 1 , the CXL switch 302 or the first VCS 320 or the second VCS 322 may coordinate the communication between the host device and the various accelerators. For example, the CXL switch 302 may be coupled to various CXL devices (e.g., the first CXL device 318 or the second CXL device 324), or coupled to various logical devices such as a single logical device (LD, e.g., the first LD 314, the second LD 316, the third LD 326, or the fourth LD 328) via multiple logical devices (MLD, e.g., MLD 312). Each CXL device and logical device may represent a corresponding accelerator or CXL device with its own corresponding CXL.io configuration space, CXL.mem memory space, and CXL.cache cache space.

[0054] Figure 4A , Figure 4B , Figure 5A , Figure 5B and Figure 5C Respective examples of a CXL host device (e.g., host device 202) coupled to an accelerator device (e.g., CXL device 204) are generally described, the accelerator device, for example, may include a memory device. In an example, the accelerator device includes a CXL controller that manages transactions with the host, and the accelerator includes a memory controller that manages transactions with a memory (MEM). The memory may include or use volatile memory, such as DRAM, SDRAM, PCRAM, RRAM, and other kinds of memory. The memory may additionally or alternatively include or use non-volatile memory, such as NAND or NOR flash memory. Although the host and accelerator devices are discussed in various examples herein as "CXL" host devices and "CXL" accelerators or "CXL" devices, other types of hosts and accelerators may be similarly used without including or using the CXL protocol.

[0055] Figure 4A An example of an accelerator device having a single back-end media controller or memory controller is generally described. The memory controller may include or use a relatively slow memory (e.g., DRAM) in terms of, for example, access or data transfer rates. Figure 4A In an example of , queuing delays on the accelerator device may occur due to over-utilization or overloading of the request path. For example, when requests arrive faster than the backend media or memory (MEM) can service the requests, the forward path or request path in the accelerator device may be overloaded. That is, the CXL controller and memory controller on the accelerator device may have high request queue utilization.

[0056] exist Figure 4AIn an example, the backward path or response path may be underutilized or relatively lightly loaded. That is, the CXL controller and the memory controller on the accelerator device may have low response queue utilization. In an example, the CXL host device may be configured to help mitigate queuing delays by monitoring the request queue of the memory controller on the accelerator device and assigning new queues to other resources when the memory controller is overutilized.

[0057] Figure 4B Examples of accelerator devices that include or use relatively fast memory (e.g., SRAM) in terms of, for example, access or data transfer rates are generally described. Figure 4B In an example of , queuing delays on the accelerator device may occur due to overutilization or overloading of the response path. For example, the forward path or request path in the accelerator device may be lightly loaded because the backend media or memory (MEM) can service the requests relatively quickly when they arrive from the CXL controller. Therefore, the request queue of the memory controller may not be fully utilized.

[0058] exist Figure 4B In an example, the backward path or response path may be overutilized due to the relatively fast backend media. Therefore, the CXL controller and memory controller on the accelerator device may have high response queue utilization. In an example, the CXL host device may be configured to help alleviate response queue overutilization by monitoring the response queue at the CXL controller and assigning new queues to other resources when the response queue at the CXL controller is overutilized.

[0059] Figure 5A , Figure 5B and Figure 5C An example of an accelerator device having multiple back-end media controllers or memory controllers is generally described. Figure 5A In an example of , the accelerator memory controller may include or use a memory device (e.g., DRAM) that is relatively slow in terms of, for example, access or data transfer rates. Figure 5A In an example of , queuing delays on the accelerator device may occur due to, for example, over-utilization or overloading of request paths to one or more memory controllers. For example, when requests arrive at one or more memory controllers faster than the corresponding back-end media or memory devices (MEM) can service the requests, the forward path or request path in the accelerator device may be overloaded. That is, the CXL controller and memory controller on the accelerator device may have high request queue utilization.

[0060] exist Figure 5AIn an example, the backward path or the response path may be underutilized or relatively lightly loaded. That is, the CXL controller and the memory controller on the accelerator device may have low response queue utilization. In an example, the CXL host device may be configured to help mitigate queuing delays by monitoring the request queues of each of the memory controllers on the accelerator device and assigning new queues to other resources when one or more of the memory controllers are overutilized.

[0061] Figure 5B An example of an accelerator device is generally described that includes multiple memory controllers configured to use corresponding memories (e.g., SRAM) that are relatively fast in terms of, for example, access or data transfer rates. Figure 5B In an example of , queuing delays on the accelerator device may occur due to, for example, overutilization or overloading of response paths in one or more memory controllers. For example, a forward path or request path in the accelerator device may be lightly loaded because the backend media or memory (MEM) can service requests relatively quickly as they arrive from the CXL controller. As a result, the request queues of the corresponding memory controllers may not be fully utilized.

[0062] exist Figure 5B In an example, the backward path or response path through one or more of the memory controllers may be overutilized due to the relatively fast backend media. Therefore, the CXL controller and the memory controller on the accelerator device may have high response queue utilization. In an example, the CXL host device may be configured to help mitigate response queue overutilization by monitoring the response queue at the CXL controller and assigning a new queue to other resources when the response queue at the CXL controller is overutilized.

[0063] Figure 5C An example of an accelerator device is generally described that includes multiple memory controllers configured to use respective memory devices that may have the same or different speed characteristics. Figure 5C In an example of , responses from each of two or more memory controllers on the accelerator device may collide or aggregate on a particular CXL response queue at the CXL controller. In this example, the response queue at each of the individual memory controllers may not be saturated, however, when multiple memory controllers push response messages to the CXL controller, the CXL controller may be oversubscribed.

[0064] In an example, a CXL host device may be configured to help mitigate response queue over-utilization by monitoring response queue activity at the CXL controller and at each of the memory controllers. For example, when multiple memory controllers may provide responses to the same CXL controller, monitoring memory controller queue utilization alone may not be sufficient.

[0065] Figure 6 An example of an accelerator device 602 including a telemetry manager 604 and a thermal manager 606 is generally described. In an example, the accelerator device 602 is a CXL accelerator device configured to communicate with one or more hosts via a CXL interface, for example, using transactions defined by the CXL.io, CXL.mem, and CXL.cache protocols. The accelerator device 602 may include a memory device that includes one or more memories, for example, memories of the same type or different types. In an example, the thermal manager 606 may receive thermal characteristic information from one or more portions of the accelerator device 602. In an example, the telemetry manager 604 is configured to receive performance metric information or DevLoad information from various components of the accelerator device 602 or components coupled to the accelerator device, and the telemetry manager 604 may be configured to receive thermal characteristic information from the thermal manager 606. The telemetry manager 604 may be configured to report DevLoad information about the accelerator device 602 or its components to one or more host devices. In an example, telemetry manager 604 may be configured to calculate or determine a DevLoad indicative metric for accelerator device 602 (or one or more components thereof) based on request and / or response path utilization and based on thermal characteristics from one or more portions of accelerator device 602 .

[0066] For ease of illustration and discussion, the example of accelerator device 602 includes a conceptual front end portion 608, a mid-end portion 610, and a back end portion 612. Portions of accelerator device 602 and its components may be configured or combined in different ways according to different implementations of accelerator device 602.

[0067] exist Figure 6In an example, the front end portion 608 may include a CXL link 618 configured to interface with a host device using a CXL PCIe PHY layer 616. The front end portion 608 may further include a CXL data link layer 620 and a CXL transport layer 622 configured to manage transactions between the accelerator device 602 and the host. In an example, the CXL transport layer 622 includes registers and operators configured to manage a CXL request queue (e.g., including one or more memory transaction requests) and a CXL response queue (e.g., including one or more memory transaction responses) for the accelerator device 602. In an example, the front end portion 608 includes a read and write manager 614. The read and write manager 614 may be configured to monitor read and write transactions and transaction status information within the accelerator device 602. For example, the read and write manager 614 may be configured to report a read / write ratio as an indicator of the DevLoad of the accelerator device 602 to the telemetry manager 604. In an example, the read-write manager 614 provides ratio information using the read-write signals 648 , which, for example, may include metrics with information about reads, writes, or both reads and writes.

[0068] In an example, the accelerator device 602 may include a memory device that includes a cache (e.g., including SRAM) and includes long-term volatile or non-volatile memory accessible via a memory controller. Figure 6 In the example of , the accelerator device 602 includes a cache memory 626 in the mid-end portion 610 of the device. The mid-end portion 610 may include a cache controller 624 configured to monitor requests from the CXL transport layer 622 and identify requests that can be satisfied using the cache memory 626.

[0069] Various complexities may arise in a CXL system. For example, CXL transactions may be based on relatively large transaction sizes (e.g., 64 bytes), while some processes may use larger granularity or smaller data sizes. Therefore, in some examples, a cache controller 624 may be included or used in the accelerator device 602 to store excess data fetched from a backend media controller or memory, such as from one or more memories in the backend portion 612 of the accelerator device 602.

[0070] In an example, the cache controller 624 is coupled to a crossbar interface or XBAR interface 628. The XBAR interface 628 can be configured to allow multiple requesters to access multiple memory controllers in parallel, such as multiple memory controllers in the back end portion 612 that includes the accelerator device 602. In an example, the XBAR interface 628 essentially provides point-to-point access between the requesters and the memory controllers, and generally provides higher performance than that available using conventional bus architectures. The XBAR interface 628 can be configured to receive responses from the back end portion 612 or receive cache hits from the cache memory 626, and deliver the responses to the front end portion 608 using a cache response queue.

[0071] exist Figure 6 In the example of , the back end portion 612 of the accelerator device 602 includes a plurality of memory controllers, including a first memory controller 630 to an Nth memory controller 634. Each of the memory controllers may have or use a corresponding memory request and response queue. Each of the memory controllers may be coupled to a corresponding medium or memory, which may include, for example, volatile or non-volatile memory. In the illustrated example, the first memory controller 630 is coupled to the first memory 632, and the Nth memory controller 634 is coupled to the Nth memory 636.

[0072] In an example, each of the multiple memory controllers in the system can manage its own respective queue. In some examples, different memory controllers can be configured to use or interface with different memories, which, for example, can have different latency characteristics. Therefore, performance optimization can include coordination of the respective queues of each memory controller. Informed coordination can be based on, for example, request and response path information of each memory controller.

[0073] In an example, a solution to the resource allocation problem described herein for distributed CXL devices or other accelerators may include identifying queues that cause or may cause a bottleneck or constraint on the functionality of a portion of a system. In an example, the queues may correspond to different types of accelerators, different types of CXL devices, which may, for example, have different device topologies. Each or more CXL devices in the system may be configured to provide a DevLoad indication signal to a system host or hosts. In response, one or more hosts may use the one or more DevLoad indication signals to help optimize queue generation across multiple devices, for example based on resource utilization on accelerators available to the hosts.

[0074] exist Figure 6In an example of the present invention, the accelerator device 602 includes a telemetry manager 604, which is configured to receive performance or DevLoad metric information from various components of the accelerator device 602 and provide a DevLoad indication signal 650 to the accelerator device 602. The DevLoad indication signal 650 can be provided to a host, a fabric manager, or other controller in a system in which the accelerator device 602 is used. In response to the DevLoad indication signal 650, the host, the fabric manager, or other controller can determine which accelerator devices 602 in the system are underutilized or overutilized. For example, based on the DevLoad indication signal 650, the host can assign new queues or threads to underutilized resources, or can dynamically adjust virtualized memory space to maximize efficiency and minimize latency.

[0075] In an example, the telemetry manager 604 may be configured to determine the DevLoad indication signal 650 using request queue utilization information from each memory controller in the backend portion 612 of the accelerator device 602. For example, the telemetry manager 604 may receive a first memory controller request queue utilization signal 638 having information about utilization characteristics of a request queue for a first memory controller 630, and the telemetry manager 604 may receive a second memory controller request queue utilization signal 640 having information about utilization characteristics of a request queue for an Nth memory controller 634. The memory controller request queue utilization signals include information about occupancy within each respective memory controller. In an example, if the request queue occupancy of a particular memory controller exceeds a predetermined occupancy threshold, the memory or media associated with the particular memory controller may be a bottleneck that may cause the request queue to be overutilized. In response to such overutilization, the host may throttle requests sent to the particular memory controller or to a particular CXL device including the particular memory controller.

[0076] The telemetry manager 604 may be configured to determine the DevLoad indication signal 650 using, for example, accelerator-level request or response queue utilization information from the front-end portion 608. For example, the telemetry manager 604 may receive a CXL request queue utilization signal 644 that includes information about the amount or number of requests received by the accelerator device 602 (e.g., via CXL.mem) and sent or pending at the mid-end portion 610, the back-end portion 612, or other portion of the accelerator device 602. In an example, the telemetry manager 604 may receive a CXL response queue utilization signal 646 that includes information about the amount or number of responses received or pending at the mid-end portion 610, the back-end portion 612, or other portion of the accelerator device 602.

[0077] In an example, information from the CXL request queue utilization signal 644 can be used to optimize a topology that includes or uses a central cache controller, such as the cache controller 624, for example, where the mid-portion 610 may become a bottleneck. In such cases, the memory controller utilization signal may not reveal information that the device forward path is over-utilized. In an example, if the CXL request queue utilization signal 644 indicates high occupancy, but the memory controller request queue utilization signal indicates low occupancy, then a subsystem in the mid-portion 610 of the accelerator device 602 (e.g., the cache controller 624 and associated components) may be determined to be a performance bottleneck.

[0078] In an example, information from CXL response queue utilization signal 646 may be used to track queue occupancy within the CXL controller. If CXL response queue utilization signal 646 indicates that the queue occupancy is greater than a specified threshold amount, then it may be determined that the response link is overutilized. In other words, it may be determined that the media or memory in backend portion 612 is fast enough to satisfy requests, but the CXL response link is a performance bottleneck.

[0079] The telemetry manager 604 may be configured to receive a thermal state signal 642 from the thermal manager 606. The thermal state signal 642 may include one or more signals indicating corresponding thermal characteristics of a portion of the accelerator device 602. In an example, the thermal state signal 642 is a composite or combined signal indicating the temperature characteristics of the accelerator device 602, such as an average value of multiple temperatures of corresponding different portions of the device. In an example, the thermal state signal 642 includes a binary signal indicating whether the thermal characteristics or temperature of any of the subsystems or portions of the accelerator device 602 exceeds a specified threshold thermal condition. In some examples, information about specific thermal characteristics of the accelerator device 602 in the thermal state signal 642 may be used by the telemetry manager 604 to indicate an immediate response, such as without regard to queue utilization information received by the telemetry manager 604. For example, if a portion of the accelerator device 602 is indicated as overheated, the DevLoad of the device may be throttled or changed immediately regardless of the queue utilization state.

[0080] The telemetry manager 604 may be configured to receive a read / write signal 648 from the read / write manager 614. The read / write signal 648 may include metrics with information about, for example, reads, writes, or both reads and writes (e.g., expressed as a ratio) performed by the accelerator device 602 or queued for use or transmission by the accelerator device within or within a specified duration. For example, the read / write signal 648 may include information about the relative amount of read information and write information contained in a CXL message transmitted using the CXL link 618. The CXL message may include, for example, a 64-byte flow control unit or FLIT, such as including multiple time slots (e.g., 4 16-byte time slots) or a 256-byte FLIT, etc. In an example, a write response (e.g., a non-data response or NDR) may be effectively encapsulated in a 64-byte CXL message or response FLIT (e.g., 1 time slot may carry 3 write responses), while the data of a particular read response (DRS) may occupy multiple time slots, such as in one or more FLITs. As an example, 2 read responses may consume 9 time slots, e.g., using 3 FLITs transmitted using CXL link 618. Thus, if the CXL response queue contains a relatively large number of write responses (e.g., greater than a specified threshold number of write responses), the responses may be issued relatively efficiently and quickly using a small number of FLITs using CXL link 618. On the other hand, read responses may take up more space and thus may take longer to transmit or may use more FLITs. Thus, it may be important to track the type of information in the CXL response queue (e.g., write responses vs. read responses) to inform how quickly the CXL response queue may be cleared.

[0081] In an example, the telemetry manager 604 may use the various input signals described above to determine the DevLoad indication signal 650. For example, the telemetry manager 604 may decode one or more of the first memory controller request queue utilization signal 638, the second memory controller request queue utilization signal 640, the thermal status signal 642, the CXL request queue utilization signal 644, the CXL response queue utilization signal 646, and the read / write signal 648, and determine a utilization status of the accelerator device 602 or a portion of the accelerator device 602 that may be reported to the host. In an example, the accelerator device 602 may report utilization status information from the telemetry manager 604, wherein each response message is sent to the host via the CXL link 618. In response, the host may optionally update or change the traffic rate of the accelerator device 602. The host-based traffic rate adjustment may be host-specific or application-specific.

[0082] Fig. 7A , Figure 7B and Figure 7CA respective example of a CXL host device (e.g., host device 202) coupled to an accelerator device 602 is generally described, which, for example, may include or use a memory device configured to use CXL. The accelerator device 602 may include, for example, a CXL controller portion (e.g., corresponding to a front-end portion 608 of the accelerator device 602), a cache controller 624 (e.g., corresponding to a mid-end portion 610 of the accelerator device 602), and a memory controller portion (e.g., corresponding to a back-end portion 612 of the accelerator device 602). Thus, an example of the accelerator device 602 may include a CXL device having a central cache controller and one or more back-end media or memory controllers.

[0083] exist Fig. 7A In an example, the cache controller may have a relatively low cache hit rate, and most requests may be sent to the memory controller by the device. In this example, the memory controller request queue may have a relatively high utilization compared to the response queue. Depending on the number of memory controllers used in the accelerator device 602, the utilization or DevLoad of the accelerator device 602 may be monitored using, for example, a memory controller request queue signal (e.g., a first memory controller request queue utilization signal 638, a second memory controller request queue utilization signal 640, etc.).

[0084] exist Figure 7B In an example of , the cache controller may have a relatively high hit rate, and most requests may be served by the cache in the middle portion 610 of the accelerator device 602. In other words, relatively few requests may need to be served by the back-end media or memory controller. In this example, due to the relatively high speed and hit rate at the cache controller, the CXL response path may be overutilized. If the load of new requests on the inlet port of the accelerator device 602 increases, the utilization of the memory controller response and request queues can be kept relatively low as long as the cache hit level is maintained. However, if the cache controller is slow, the cache controller may throttle the CXL request queue, and the CXL request queue utilization signal 644 may indicate overutilization. In this example, the utilization or DevLoad of the accelerator device 602 may be monitored using, for example, the CXL request queue utilization signal 644 and the CXL response queue utilization signal 646.

[0085] exist Figure 7C In the example of , the accelerator device 602 may include or use multiple cache controllers. Figure 5CIn an example, responses from cache controllers may potentially conflict or accumulate on a particular CXL response queue at the CXL controller. In this example, the response queue at each of the individual memory controllers or cache controllers may be unsaturated, however, when multiple cache controllers push response messages to the CXL controller, the CXL controller may be oversubscribed. In an example, even when one of the cache controllers has a high hit rate and the other cache controller has a low hit rate, cache responses may accumulate on the CXL response queue and cause a bottleneck. In this example, the utilization or DevLoad of the accelerator device 602 may be monitored using, for example, the CXL response queue utilization signal 646 and one or more of the first memory controller request queue utilization signal 638, the second memory controller request queue utilization signal 640, or other memory controller request queue utilization signals.

[0086] Fig. 7A , Figure 7B and Figure 7C 602 may provide a DevLoad indication signal 650 to the host, and the host may determine which mitigation measures to take. The utilization of multiple accelerator device internal resources may be tracked and monitored, and the multiple accelerator device internal resources may be analyzed together to provide the DevLoad indication signal 650.

[0087] According to various examples, the DevLoad indication signal 650 may indicate a classification or association of different levels of device resource loading or occupancy of the accelerator device 602 relative to, for example, light loading (e.g., corresponding to low loading or underutilization of at least a portion of the device), optimal loading, medium loading, and heavy loading (e.g., corresponding to high loading or overutilization of at least a portion of the device). In some examples, such stratification of device occupancy or loading may correspond to standardized classifications under the CXL specification. Different request and response utilization signals may correspond to different DevLoad classification indication information in the DevLoad indication signal 650. The following table provides some examples of the relationship between the information in the DevLoad indication signal 650 and various request and response queue hierarchies.

[0088]

[0089]

[0090] Table 1. Information of the DevLoad indication signal 650 when the memory controller request queue utilization signal indicates light utilization.

[0091]

[0092]

[0093] Table 2. Information of the DevLoad indication signal 650 when the memory controller request queue utilization signal indicates optimal utilization.

[0094]

[0095] Table 3. Information of the DevLoad indication signal 650 when the memory controller request queue utilization signal indicates medium utilization.

[0096]

[0097] Table 4. Information of the DevLoad indication signal 650 when the memory controller request queue utilization signal indicates high utilization.

[0098] Figure 8 A first method 800 is generally illustrated that may include determining a device loading metric for an accelerator device (eg, the example accelerator device 602 ) in a CXL-based system.

[0099] At operation 802, the first method 800 may include receiving a command from a host device at a first accelerator device, such as the accelerator device 602. In an example, operation 802 may include receiving a command at a front end portion 608 of the accelerator device 602 using a CXL link 618. The command may include, for example, a write command, a read command, or other instructions for an operation to be performed by the first accelerator device. In an example, the first accelerator device includes a telemetry manager, such as the telemetry manager 604. The telemetry manager may be configured to monitor or receive thermal information, transaction status information, or other information about one or more portions of the accelerator device, and in response, provide an accelerator device load indication signal or DevLoad indication signal 650 to the host device. The host device may use the load indication signal to throttle or control future instructions provided to the first accelerator device.

[0100] At operation 804, the first method 800 may include receiving, at a telemetry manager of the first accelerator device, a queue utilization signal indicating a volume of transaction request messages received by the first accelerator device or response messages sent from the first accelerator device. In an example, the volume of transaction messages includes a count or number of request messages or response messages exchanged with the host device or in response to a command from the host device.

[0101] At operation 806 , the first method 800 may include determining, at a telemetry manager of the first accelerator device, a device load metric of the first accelerator device based on the queue utilization signal. For example, the device load metric may be based on the queue utilization signal received by the telemetry manager at operation 804 .

[0102] At operation 808, the first method 800 may include providing a first control signal to the host device based on the device loading metric determined at operation 806. In an example, the first control signal includes information about utilization or loading of the first accelerator device and its availability or capacity to process subsequent commands or requests from the host device.

[0103] Fig. 9 A second example 900 is generally described that may include using information about device load metrics of multiple accelerator devices to select a device for subsequent operation. The second example 900 may include or use a host device coupled to at least a first accelerator device and a second accelerator device, such as using a CXL link, and each of the accelerator devices is a separate instance of, for example, the accelerator device 602. The device load metrics may include information about occupancy of the accelerator device (e.g., in terms of queue utilization in one or more portions of the accelerator device), information about thermal characteristics of the accelerator device, or other information about the accelerator device indicating its capacity to process additional requests (e.g., read or write requests) from the host device.

[0104] At block 902, the second example 900 may include receiving a first device utilization signal or a first DevLoad signal from a first accelerator device using a host device. At block 904, the second example 900 may include receiving a second device utilization signal or a second DevLoad signal from a second accelerator device using a host device.

[0105] At block 908, the second example 900 may include using the host device to select a specific accelerator device to receive subsequent commands from the host device. The host device may use information from the first DevLoad signal and the second DevLoad signal and determine which of the first accelerator device and the second accelerator device can best accommodate additional requests or loads from the host device. In an example, the first DevLoad signal and the second DevLoad signal may indicate that each of the first accelerator device and the second accelerator device is overutilized, and in response, the host device may select a different third accelerator device to receive later commands. At block 910, the second example 900 may include using the host device to provide subsequent commands to the specific accelerator device selected at block 908.

[0106] Fig.10A block diagram illustrating an example machine 1000, which can be used, implemented in, or by the machine to implement any one or more techniques (e.g., methods) discussed herein. As described herein, an example may include logic or multiple components or mechanisms in the machine 1000, or may be operated by logic or multiple components or mechanisms in the machine 1000. A circuit system (e.g., a processing circuit system) is a collection of circuits implemented in a tangible entity of the machine 1000, which includes hardware (e.g., simple circuits, gates, logic, etc.). Circuit system membership (e.g., belonging to a host-side device or process, or belonging to an accelerator-side device or process) can change flexibly over time. The circuit system includes components that can perform specified operations individually or in combination when operating. In an example, the hardware of the circuit system can be designed in an immutable manner to implement specific operations (e.g., hard-wired), such as using the accelerator logic 222 or using its specific command execution unit. In an example, the hardware of the circuit system may include variably connected physical components (e.g., command execution units, transistors, simple circuits, etc.), which include machine-readable media that are physically modified (e.g., magnetic, electrical, removable placement of fixed centralized particles, etc.) to encode instructions for specific operations. When the physical components are connected, the basic electrical properties of the hardware composition are changed from insulators to conductors, or from conductors to insulators. The instructions enable embedded hardware (e.g., execution units or loading mechanisms) to generate circuit system components in hardware via variable connections to perform parts of specific operations when in operation. Therefore, in an example, the machine-readable media element is part of the circuit system or is coupled to other components of the circuit system in a communicative manner when the device is operating. In an example, any of the physical components can be used in more than one component in more than one circuit system. For example, under operation, the execution unit can be used in a first circuit of a first circuit system at one point in time, and reused by a second circuit in the first circuit system or reused by a third circuit in the second circuit system at a different time.

[0107] In alternative embodiments, the machine 1000 may operate as a standalone device or may be connected (e.g., networked) to other machines. In a networked deployment, the machine 1000 may operate in the capacity of a server machine, a client machine, or both in a server-client network environment. In an example, the machine 1000 may act as a peer machine in a peer-to-peer (P2P) (or other distributed) network environment. The machine 1000 may be a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile phone, a network appliance, a network router, a switch or a bridge, or any machine capable of executing (sequentially or otherwise) instructions specifying actions to be taken by the machine. In addition, although only a single machine is described, the term "machine" should also be construed to include any collection of machines that individually or collectively execute an instruction set (or multiple instruction sets) to perform any one or more of the methods discussed herein (e.g., cloud computing, software as a service (SaaS), other computer cluster configurations).

[0108] Any one or more of the components of the machine 1000 may include or use one or more instances of the host device 202 or the CXL device 204 or other components in or attached to the computing system 100. The machine 1000 (e.g., a computer system) may include a hardware processor 1002 (e.g., a host processor 214, accelerator logic 222, a central processing unit (CPU), a graphics processing unit (GPU), a hardware processor core, or any combination thereof), a main memory 1004, a static memory 1006 (e.g., memory or storage for firmware, microcode, basic input-output (BIOS), unified extensible firmware interface (UEFI), etc.), and a mass storage device 1008 or a stack of memory dies, a hard drive, a tape drive, a flash storage device, or other block devices, some or all of which may communicate with each other via an interconnect 1030 (e.g., a bus). The machine 1000 may further include a display device 1010, an alphanumeric input device 1012 (e.g., a keyboard), and a user interface (UI) navigation device 1014 (e.g., a mouse). In an example, the display device 1010, the input device 1012, and the UI navigation device 1014 may be a touch screen display. The machine 1000 may additionally include a mass storage device 1008 (e.g., a drive unit), a signal generating device 1018 (e.g., a speaker), a network interface device 1020, and one or more sensors 1016, such as a global positioning system (GPS) sensor, a compass, an accelerometer, or other sensors. The machine 1000 may include an output controller 1028, such as a serial (e.g., universal serial bus (USB), parallel or other wired or wireless (e.g., infrared (IR), near field communication (NFC), etc.) connection to communicate with or control one or more peripheral devices (e.g., printers, card readers, etc.).

[0109] The hardware processor 1002, main memory 1004, static memory 1006, or a register of a mass storage device 1008 may be or include a machine-readable medium 1022 on which is stored one or more sets of data structures or instructions 1024 (e.g., software) embodying or used by any one or more of the techniques or functions described herein. The instructions 1024 may also reside, completely or at least partially, within any of the registers of the hardware processor 1002, main memory 1004, static memory 1006, or a mass storage device 1008 during execution thereof by the machine 1000. In an example, one or any combination of the hardware processor 1002, main memory 1004, static memory 1006, or a mass storage device 1008 may constitute the machine-readable medium 1022. Although machine-readable medium 1022 is illustrated as a single medium, the term “machine-readable medium” may include a single medium or multiple media (eg, a centralized or distributed database, or associated caches and servers) configured to store one or more instructions 1024 .

[0110] The term "machine-readable medium" may include any medium capable of storing, encoding, or carrying instructions for execution by the machine 1000 and causing the machine 1000 to perform any one or more techniques of the present disclosure, or any medium capable of storing, encoding, or carrying data structures used by or associated with such instructions. Non-limiting machine-readable media examples may include solid-state memory, optical media, magnetic media, and signals (e.g., radio frequency signals, other photon-based signals, sound signals, etc.). In an example, a non-transitory machine-readable medium includes a machine-readable medium having a plurality of particles that have a constant (e.g., stationary) mass and are therefore composed of matter. Thus, a non-transitory machine-readable medium is a machine-readable medium that does not include a transient propagating signal. Specific examples of non-transitory machine-readable media may include: non-volatile memory, such as semiconductor memory devices (e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)) and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0111] In an example, the information stored or otherwise provided on the machine-readable medium 1022 may represent instructions 1024, such as the instructions 1024 themselves or a format from which the instructions 1024 may be derived. Such a format from which the instructions 1024 may be derived may include source code, encoded instructions (e.g., in compressed or encrypted form), packaged instructions (e.g., split into multiple packets), etc. The information representing the instructions 1024 in the machine-readable medium 1022 may be processed by the processing circuitry into instructions to implement any of the operations discussed herein. For example, deriving the instructions 1024 from the information (e.g., processed by the processing circuitry) may include: compiling (e.g., from source code, object code, etc.), interpreting, loading, organizing (e.g., dynamically or statically linking), encoding, decoding, encrypting, decrypting, packaging, unpacking, or otherwise manipulating the information into instructions 1024.

[0112] In an example, the export of instructions 1024 may include (e.g., by processing circuitry) the assembly, compilation, or interpretation of information to create instructions 1024 from some intermediate or pre-processed format provided by machine-readable medium 1022. When the information is provided in multiple parts, the information may be combined, unpacked, and modified to create instructions 1024. For example, the information may be in multiple compressed source code data packets (or object code, or binary executable code, etc.) on one or several remote servers. The source code data packets may be encrypted when transmitted over the network, and decrypted, decompressed, assembled (e.g., linked) as necessary, and compiled or interpreted at the local machine (e.g., into a stand-alone executable library, etc.), and executed by the local machine.

[0113] The instructions 1024 may further be transmitted or received over the communication network 1026 using a transmission medium via the network interface device 1020 using any of a number of transmission protocols (e.g., frame relay, Internet Protocol (IP), transmission control protocol (TCP), user datagram protocol (UDP), hypertext transfer protocol (HTTP), etc.). Example communication networks may include a local area network (LAN), a wide area network (WAN), a packet data network (e.g., the Internet), a mobile telephone network (e.g., a cellular network), a plain old telephone (POTS) network, and a wireless data network (e.g., a wireless network known as a wireless network). The Institute of Electrical and Electronics Engineers (IEEE) 802.11 series of standards, known as 16 family of standards), IEEE 802.15.4 family of standards, peer-to-peer (P2P) networks, and the like. In an example, the network interface device 1020 may include one or more physical jacks (e.g., Ethernet, coaxial, or telephone jacks) or one or more antennas to connect to the network 1026. In an example, the network interface device 1020 may include multiple antennas to wirelessly communicate using at least one of single-input multiple-output (SIMO), multiple-input multiple-output (MIMO), or multiple-input single-output (MISO) technology. The term "transmission media" should be taken to include any intangible media that can store, encode, or carry instructions for execution by the machine 1000 and includes digital or analog communication signals, or other intangible media used to facilitate the communication of such software. Transmission media is machine-readable media.

[0114] The above detailed description includes references to the accompanying drawings, which form a part of the detailed description. The drawings show by way of illustration specific embodiments in which the present invention can be practiced. These embodiments are also referred to herein as "examples". Such examples may include elements other than those shown or described. However, the inventors also contemplate examples in which only those elements shown or described are provided. In addition, the inventors also contemplate examples (or one or more aspects thereof) using any combination or arrangement of those elements shown or described relative to a specific example (or one or more aspects thereof) or relative to other examples (or one or more aspects thereof) shown or described herein.

[0115] In this document, as is common in patent documents, the term "one" or "an" is used to include one or more than one, which is independent of any other examples or uses of "at least one" or "one or more". In this document, unless otherwise indicated, the term "or" is used to refer to a non-exclusive or, so that "A or B" may include "A but not B", "B but not A" and "A and B". In the appended claims, the terms "including" and "in which" are used as the plain English equivalents of the corresponding terms "including" and "wherein". Moreover, in the appended claims, the terms "including" and "comprising" are open, that is, systems, devices, objects or processes containing elements other than those listed after such terms in the claims are still considered to be within the scope of the claims. In addition, in the appended claims, the terms "first", "second" and "third" etc. are used only as markers and are not intended to impose numerical requirements on their objects.

[0116] The above description is intended to be illustrative and not restrictive. For example, the examples described above (or one or more aspects thereof) may be used in combination with each other. For example, a person of ordinary skill in the art may use other embodiments after consulting the above description. It should be understood that the embodiments will not be used to interpret or limit the scope or meaning of the claims. Moreover, in the above specific embodiments, various features may be grouped together to simplify the disclosure. This should not be interpreted as wishing that unclaimed public features are necessary for any claim. In fact, the subject matter of the present invention may be less than all features of a specific disclosed embodiment. Therefore, the attached claims are hereby incorporated into the specific embodiments, wherein each claim exists independently as a separate embodiment, and it is considered that such embodiments may be combined with each other in various combinations or arrangements. The scope of the present invention should be determined by reference to the attached claims and the full scope of equivalents authorized by the claims.

Claims

1. A method comprising: receiving, at the first accelerator device, a command from a host device; At the first accelerator device: receiving a queue utilization signal indicating a volume of transaction request messages received by the first accelerator device or response messages sent from the first accelerator device based on the command from the host device; determining a device loading metric for the first accelerator device based on the queue utilization signal; and A first control signal is provided to the host device, wherein the first control signal includes information regarding the device loading metric of the first accelerator device. 2 . The method of claim 1 , comprising distributing, at the host device, subsequent commands to the first accelerator device and a second accelerator device based on the first control signal from the first accelerator device.

3. The method of claim 2, comprising receiving, at the host device, a second control signal having information about a device load metric of the second accelerator device; and Wherein allocating the subsequent command is based on the first control signal and the second control signal. 4 . The method of claim 1 , wherein receiving the command from the host device comprises receiving the command using a compute express link (CXL) interconnect, and wherein providing the first control signal to the host device comprises using the CXL interconnect. 5 . The method of claim 4 , wherein at least a portion of the first control signal is provided to the host device together with each flow control unit FLIT transmitted from the first accelerator device to the host device. 6 . The method of claim 4 , wherein receiving the queue utilization signal comprises receiving a CXL response queue utilization signal indicating an amount of transactions queued waiting to be transmitted from the first accelerator device to the host using the CXL interconnect. 7 . The method of claim 4 , wherein receiving the queue utilization signal comprises receiving a CXL request queue utilization signal indicating a number of transactions queued for further processing by a compute resource of the first accelerator device. 8 . The method of claim 7 , wherein the CXL request queue utilization signal includes information regarding utilization of a cache controller on the first accelerator device. 9 . The method of claim 1 , wherein receiving the queue utilization signal comprises receiving a memory controller request queue utilization signal from a memory controller that comprises a portion of the first accelerator device.

10. The method of claim 1, wherein receiving the queue utilization signal comprises receiving a memory controller response queue utilization signal from a memory controller comprising a portion of the first accelerator device.

11. The method of claim 1 , comprising determining a read / write ratio of transactions processed by the first accelerator device, the ratio being based on a number of data response (DRS) messages and a number of non-data response (NDR) messages queued for transmission from the first accelerator device to the host device; and Wherein determining the device loading metric comprises using the determined read / write ratio.

12. The method of claim 1, further comprising, at a telemetry manager of the first accelerator device, receiving a thermal status signal indicating a temperature of a portion of the first accelerator device, and determining the device load metric for the first accelerator device based on the thermal status signal and the queue utilization signal.

13. The method of claim 1, further comprising, at the host device: receiving the control signal from the first accelerator device and receiving at least one other control signal from a second accelerator device; and Based on the control signal, a particular one of the first accelerator device and the second accelerator device is selected to receive a subsequent command.

14. The method of claim 1, further comprising, at the host device: receiving the control signal from the first accelerator device; Based on the control signal, classifying the first accelerator device as underutilized, overutilized, or optimally utilized; and Based on the classification of the first accelerator device, the first accelerator device or a different accelerator device coupled to the host device is selected to execute a subsequent command.

15. The method of claim 14, wherein classifying the first accelerator device comprises using information from the control signal regarding one or more of a request path loading state, a response path loading state, a read / write transaction ratio of the first accelerator device, and a thermal state of the first accelerator device.

16. A system comprising: a host device coupled to the plurality of accelerator devices using an interconnect; as well as A first accelerator device among the plurality of accelerator devices, the first accelerator device comprising: a first controller configured to manage transactions with the host device via the interconnect; a memory controller configured to manage transactions with the memory; as well as A telemetry manager is configured to receive at least one of a request queue utilization signal and a response queue utilization signal from the first controller or the memory controller, and based on the at least one utilization signal, provide a device utilization-indicating DevLoad signal to the host device via the first controller.

17. The system of claim 16, wherein the first accelerator device includes a cache controller coupled to a cache memory, and wherein the telemetry manager is configured to provide the device utilization-indicative signal based on information regarding utilization of the cache controller.

18. The system of claim 16, wherein the first accelerator device includes a thermal manager configured to receive temperature information about at least a portion of the first accelerator device, and wherein the telemetry manager is configured to provide information about the temperature of the first accelerator device in the DevLoad signal.

19. The system of claim 16, wherein the host device is coupled to the plurality of accelerator devices using a compute express link (CXL) interconnect.

20. The system of claim 19, wherein the first controller is configured to include information about the DevLoad signal in each FLIT transmitted to the host device using the interconnect.

Images