Maintaining connection with CXL host upon reset

Abstract

The invention relates to maintaining connection with a CXL host when reset. A memory device may be coupled to a host device using a compute fast link (CXL) interconnect. The memory device may include host interface circuitry and processing logic circuitry, such as subsystem manager circuitry. The host interface circuitry of the memory device may be configured to operate in one of an autonomous mode and a distribution mode. In the autonomous mode, the host interface circuit is configured to maintain a connection between the host device and the memory device when the subsystem manager circuit of the memory device is unavailable, such as when the subsystem manager circuit experiences a reset routine. In the distribution mode, the host interface circuitry is configured to allow communication between the host device and the subsystem manager circuitry of the memory device.

Description

Technical Field

[0001] The present disclosure relates to maintaining a connection with a CXL host across a reset. Background Art

[0002] Memory devices used in computers or other electronic devices can be classified as volatile and non-volatile memory. Volatile memory requires power to maintain its data and includes random access memory (RAM), dynamic random access memory (DRAM), or synchronous dynamic random access memory (SDRAM), etc. Non-volatile memory can save stored data when not powered, and includes flash memory, read-only memory (ROM), electrically erasable programmable ROM (EEPROM), static RAM (SRAM), erasable programmable ROM (EPROM), resistance variable memory, phase change memory, storage class memory, resistance random access memory (RRAM) and magnetoresistive random access memory (MRAM), etc. Persistent memory is an architectural attribute of the system, where the data stored in the media is available after a system reset or power cycle. In some examples, non-volatile memory media can be used to build a system with a persistent memory model.

[0003] The memory device may be coupled to a host (e.g., a host computing device) to store data, commands, and / or instructions for use by the host when the computer or electronic system is operating. For example, data, commands, and / or instructions may be transferred between the host and the memory device during operation of the computing or other electronic system.

[0004] Various protocols or standards may be applied to facilitate communication between a host and one or more other devices (such as a memory buffer, accelerator, or other input / output device). In an example, an out-of-order protocol such as Compute Express Link (CXL) may be used to provide high bandwidth and low latency connectivity. Summary of the invention

[0005] One aspect of the present disclosure relates to a memory device comprising: a host interface circuit configured to receive commands from a host device via a compute express link (CXL) interconnect; and a processing logic circuit system configured to control transactions with the memory device; wherein the host interface circuit is configured to inhibit communications with the host device and maintain a connection with the host device when the processing logic circuit system experiences a reset initiated internally by the memory device.

[0006] Another aspect of the disclosure relates to a method, comprising: detecting a reset condition at a memory device, wherein the memory device is coupled to a host device using a compute fast link (CXL) interconnect; in response to the reset condition: during a first phase, quiescing transactions with the host device, using host interface circuitry of the memory device to maintain a connection with the host device, and resetting processing logic circuitry of the memory device; during a second phase following the first phase, using the processing logic circuitry of the memory device to maintain the connection with the host device while resetting the host interface circuitry of the memory device; and during a third phase following the second phase, using the host interface circuitry of the memory device and the processing logic circuitry of the memory device to manage transactions with the host device.

[0007] Yet another aspect of the present disclosure relates to a method for operating a peripheral device coupled to a host device using a compute express link (CXL) interconnect, the method comprising: operating a host interface circuit of the peripheral device in one of an autonomous mode and a distribution mode, wherein in the autonomous mode, the host interface circuit is configured to maintain a connection between the host device and the peripheral device and inhibit communication between the host device and the peripheral device when a processing logic circuit system of the peripheral device is unavailable, and wherein in the distribution mode, the host interface circuit is configured to allow communication between the host device and the processing logic circuit system of the peripheral device. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] To easily identify the discussion of any particular element or act, the most significant digit or digits first in a reference number refers to the figure number in which that element is first introduced.

[0009] Figure 1 A block diagram of an example computing system including a host and memory devices is generally shown.

[0010] Figure 2 An example of a Compute Express Link (CXL) system is generally shown.

[0011] Figure 3 An example of a CXL system implementing a virtual hierarchy for managing transactions is generally shown.

[0012] Figure 4 An example of a CXL device with host interface circuitry and subsystem manager circuitry is generally shown.

[0013] Figure 5 An example of a first timing diagram for resetting a portion of a CXL device is shown.

[0014] Figure 6 An example of a second timing diagram for resetting a portion of a CXL device is shown.

[0015] Figure 7 An example of a method for resetting at least a portion of a CXL device is generally shown.

[0016] Figure 8 A block diagram illustrating an example machine with which, in which, or by which any one or more of the techniques discussed herein may be implemented. DETAILED DESCRIPTION

[0017] Compute Express Link (CXL) is an open standard interconnect configured for high-bandwidth, low-latency connectivity between host devices and other devices, such as accelerators, memory buffers, and other I / O devices. CXL is designed to facilitate high-performance computing workloads by supporting heterogeneous processing and memory systems. CXL implements coherency and memory semantics on top of PCI Express (PCIe)-based I / O semantics to optimize performance.

[0018] In some examples, CXL is used in applications such as artificial intelligence, machine learning, analytics, cloud infrastructure, edge computing devices, communication systems, and elsewhere. Data processing in such applications can use a variety of scalar, vector, matrix, and spatial architectures that can be deployed in CPUs, GPUs, FPGAs, smart NICs, or other accelerators that can be coupled using CXL links.

[0019] CXL supports dynamic multiplexing using a set of protocols including input / output (CXL.io, based on PCIe), cache (CXL.cache), and memory (CXL.memory or CXL.mem) semantics. In an example, CXL can be used to maintain a unified, consistent memory space between a CPU (e.g., a host device or host processor) and any memory on an attached CXL device. This configuration allows the CPU and CXL devices to share resources and operate on the same memory region to improve performance, reduce data movement, and reduce software stack complexity. In an example, the CPU is primarily responsible for maintaining or managing consistency in a CXL environment. Therefore, CXL can be utilized to help reduce device cost and complexity, as well as the overhead traditionally associated with consistency across I / O links.

[0020] CXL runs on top of the PCIe PHY and provides full interoperability with PCIe. In an example, a CXL device starts link training at the PCIe Gen 1 data rate and negotiates CXL as its operating protocol if its link partner supports CXL (e.g., using the alternate protocol negotiation mechanism defined in the PCIe 5.0 specification). As a result, devices and platforms can more easily adopt CXL by leveraging the PCIe infrastructure without having to design and validate PHYs, channels, channel extension devices, or other upper layers of PCIe.

[0021] In an example, CXL supports single-stage switching to enable fan-out to multiple devices. This enables multiple devices in a platform to be migrated to CXL while maintaining CXL's backward compatibility and low latency characteristics. In an example, CXL can provide a standardized computing fabric that supports multiple logical devices (MLDs) and pooling of a single logical device, such as using a CXL switch connected to several host devices or nodes (e.g., root ports). This feature enables servers to pool resources, such as accelerators and / or memory that can be assigned based on workload. For example, CXL can help facilitate resource allocation or dedicating and release. In an example, CXL can help allocate and de-allocate memory to various host devices as needed. This flexibility helps designers avoid over-provisioning while ensuring the best performance.

[0022] Some of the compute-intensive applications and operations mentioned herein may require or use large data sets. Memory devices storing such data sets may be configured for low latency and high bandwidth and persistence. One problem with load-store interconnect architectures includes ensuring persistence. CXL can help solve this problem using architectural flows of software and standard memory management interfaces, for example, enabling the migration of persistent memory from a controller-based approach to direct memory management.

[0023] The present inventors have recognized that a problem to be solved includes facilitating an internal device reset of a CXL device, including maintaining a connection between the CXL device and a host during a device reset. A CXL device reset may be triggered in various ways, such as to cause a device hardware reset or to load new device firmware. For example, a device reset may be requested by a host device in the CXL device, or may be initiated internally in the CXL device. For example, a firmware update may require a device reset, or firmware crash recovery may require a device reset. A CXL device reset may be used to restart or reboot a CXL device, such as to load new firmware or to recover the device from a serious fault or error.

[0024] The inventors have recognized that problems may include maintaining a connection between a host device and a CXL device during a reset operation. For example, if a device reset causes all parts or systems of the CXL device to shut down, the connection may be lost. In this example, transactions in the CXL bus may not be acknowledged or responded to, and thus commands from the host may be discarded, or expected responses from the CXL device may never arrive. As a result, the connection between the host and the CXL device may be lost, and reestablishing the connection may result in a loss of time and may require user intervention.

[0025] The inventors have recognized that during a CXL device reset, it may be desirable or expected to maintain a connection between the CXL device and a host device. Thus, in an example that includes a reset operation in response to a reset trigger or detected reset condition, transactions on the CXL bus or interconnect may not be disturbed and events or commands received at the CXL device from the host may be acknowledged and handled. The inventors have recognized that a solution to these and other problems may include or use a CXL device having a host interface circuit and a subsystem manager circuit. The host interface circuit may include or control interface logic that manages host commands, host resets, and sideband communications with the host. The subsystem manager circuit may include other logic for managing host and sideband transactions, and may include or control device specific logic, such as one or more circuits or components for the CXL device.

[0026] Depending on the operating mode of the CXL device, the host interface circuit or the subsystem manager circuit may be configured to manage transactions with the host device. In other words, a solution to the reset problem may include a multi-processor system that allows a CXL device reset to occur while maintaining a CXL link between the CXL device and the host. In some examples, the solution allows handling of PCIe and CXL resets, and accommodating sideband events to maintain a connection between the CXL device and the host, such as when at least a portion of the CXL device undergoes a reset operation.

[0027] In an example, the solution includes a CXL device configured to operate in multiple modes, including an autonomous mode and a distributed mode. In the autonomous mode, the subsystem manager circuit may be unavailable due to an ongoing reset, and the CXL device is configured to operate firmware logic using the host interface circuit to manage the PCIe and CXL.io interfaces and the sideband interface. For example, when the subsystem manager circuit is unavailable, the host interface circuit may apply back pressure on the CXL bus so that the CXL device appears to be temporarily unavailable but still connected to the host. In the distributed mode or normal operating mode, the subsystem manager circuit may be used to handle all transactions with the host.

[0028] In an example, the CXL device may be configured to operate in the distribution mode by default, however, the subsystem manager circuitry may include logic for detecting or determining when the autonomous mode is to be used. In an example, the subsystem manager circuitry may be configured to control or instruct the host interface circuitry to enter the autonomous mode under certain circumstances. For example, the CXL device may be caused to enter or use the autonomous mode in circumstances such as (1) cold start or initial host initialization of the CXL device, (2) receiving a host-initiated reset or reset command, or (3) a CXL device-initiated reset, such as in response to a triggering event such as a critical fault or other device condition.

[0029] Figure 1 A block diagram of an example of a computing system 100 is generally shown including a host device 102 and a memory system 104. The host device 102 includes a central processing unit (CPU) or processor 110 and a host memory 108. In an example, the host device 102 may include a host system such as a personal computer, a desktop computer, a digital camera, a smart phone, a memory card reader, and / or a device with Internet of Things capabilities, as well as various other types of hosts, and may include a memory access device, such as the processor 110. The processor 110 may include one or more processor cores, a system of parallel processors, or other CPU arrangements.

[0030] The memory system 104 includes a controller 112, a buffer 114, a cache 116, and a first memory device 118. The first memory device 118 may include, for example, one or more memory modules (e.g., a single in-line memory module, a dual in-line memory module, etc.). The first memory device 118 may include volatile memory and / or non-volatile memory, and may include a multi-chip device including one or more different memory types or modules. In an example, the computing system 100 includes a second memory device 120 that interfaces with the memory system 104 and the host device 102.

[0031] Host device 102 may include a system backplane and may include a number of processing resources (e.g., one or more processors, microprocessors, or some other type of control circuitry). Computing system 100 may optionally include separate integrated circuits for host device 102, memory system 104, controller 112, buffer 114, cache 116, first memory device 118, second memory device 120, any one or more of which may include respective chiplets that may be connected together and used together. In an example, computing system 100 includes a server system and / or a high performance computing (HPC) system and / or a portion thereof. Although Figure 1The examples shown in illustrate a system having a von Neumann architecture, but embodiments of the present disclosure may be implemented in a non-von Neumann architecture, which may not include one or more components typically associated with a von Neumann architecture (e.g., a CPU, an ALU, etc.).

[0032] In an example, the first memory device 118 may provide main memory for the computing system 100, or the first memory device 118 may include auxiliary memory or storage for use by the computing system 100. In an example, the first memory device 118 or the second memory device 120 includes one or more arrays of memory cells (e.g., volatile and / or non-volatile memory cells). For example, the array may be a flash array with a NAND architecture. The embodiments are not limited to a particular type of memory device. For example, the memory device may include RAM, ROM, DRAM, SDRAM, PCRAM, RRAM, and flash memory, etc.

[0033] In embodiments where the first memory device 118 includes persistent or non-volatile memory, the first memory device 118 may include a flash memory device, such as a NAND or NOR flash memory device. The first memory device 118 may include other non-volatile memory devices, such as non-volatile random access memory devices (e.g., NVRAM, ReRAM, FeRAM, MRAM, PCM), memory devices such as ferroelectric RAM devices including ferroelectric capacitors that may exhibit hysteresis characteristics, 3-D cross point (3D XP) memory devices, etc., or a combination thereof.

[0034] In an example, the controller 112 includes a media controller, such as a non-volatile memory express (NVMe) controller. The controller 112 may be configured to perform operations such as copying, writing, reading, error correction, etc. for the first memory device 118. In an example, the controller 112 may include dedicated circuitry and / or instructions for performing various operations. That is, in some embodiments, the controller 112 may include circuitry and / or may be configured to execute instructions to control the movement of data and / or addresses associated with the data, such as in the buffer 114, the cache 116, and / or the first memory device 118 or the second memory device 120.

[0035] In an example, at least one of the processor 110 and the controller 112 includes a command manager (CM) for the memory system 104. The CM may, for example, receive a read command for a particular logical row address in the first memory device 118 or the second memory device 120 from the host device 102. In some examples, the CM may determine that the logical row address is associated with the first row based at least in part on a pointer stored in a register of the controller 112. In an example, the CM may receive a write command for the logical row address from the host device 102, and the write command may be associated with the second data. In some examples, the CM may be configured to issue an access command associated with the first memory device 118 or the second memory device 120 to the non-volatile memory between issuing the read command and the write command. In some examples, the CM may issue an access command associated with the first memory device 118 or the second memory device 120 to the non-volatile memory between issuing the read command and the write command.

[0036] In an example, buffer 114 includes data buffer circuitry that includes, for example, an area of ​​physical memory used to temporarily store data while it is being moved from one location to another. Buffer 114 may include a first-in, first-out (FIFO) buffer in which the oldest (e.g., first-in) data is processed first. In some embodiments, buffer 114 includes a hardware shift register, a circular buffer, or a list.

[0037] In an example, the cache 116 includes an area of ​​physical memory for temporarily storing specific data that may be used again. The cache 116 may include a pool of data entries. In some examples, the cache 116 may be configured to operate according to a write-back policy, in which data is written to the cache without being concurrently written to the first memory device 118. Therefore, in some embodiments, data written to the cache 116 may not have a corresponding data entry in the first memory device 118.

[0038] In an example, the controller 112 may receive write requests related to the cache 116 (e.g., from the host device 102) and cause data associated with each of the write requests to be written to the cache 116. In some examples, the controller 112 may receive write requests at a rate of thirty-two (32) gigatransfers (GTs) per second, e.g., according to or using the CXL protocol. The controller 112 may similarly receive read requests and cause data stored in, for example, the first memory device 118 or the second memory device 120 to be retrieved and written to, for example, the host device 102 via the interface 106.

[0039] In an example, the interface 106 may include any type of communication path, bus, or the like that allows information to be transferred between the host device 102 and the memory system 104. Non-limiting examples of the interface may include a peripheral component interconnect (PCI) interface, a peripheral component interconnect express (PCIe) interface, a serial advanced technology attachment (SATA) interface, and / or a micro serial advanced technology attachment (mSATA) interface, etc. In an example, the interface 106 includes a PCIe5.0 interface that complies with the Compute Express Link (CXL) protocol standard. Therefore, in some embodiments, the interface 106 supports a transfer speed of at least 32GT / s.

[0040] As similarly described elsewhere herein, CXL is a high-speed central processing unit (CPU)-to-device or CPU-to-memory interconnect designed to enhance computing performance. CXL technology maintains memory coherency between the CPU memory space (e.g., host memory 108) and the memory on the attached device or accelerator (e.g., first memory device 118 or second memory device 120), which allows resource sharing to improve performance, reduce software stack complexity, and reduce overall system cost. CXL is designed as an industry open standard interface for high-speed communications as accelerators are increasingly used to supplement CPUs to support emerging data-rich and compute-intensive applications such as artificial intelligence and machine learning.

[0041] Figure 2 Generally, an example of a CXL system 200 is shown that connects a host device 202 and a CXL device 204 using a bus system including a CXL link bus 206 and a system management bus 208. In the example, the host device 202 includes or corresponds to the host device 102 and the CXL device 204 includes or corresponds to a host device 102. Figure 1 1. The memory system command manager (CM) may include a portion of the host device 202 or the CXL device 204.

[0042] In an example, the system management bus 208 (eg, corresponding to the Figure 1 The system management bus 208 (part of the interface 106 of the example of ) is configured to support main-band or side-band communication between the host device 202 and the CXL device 204. The system management bus 208 may use the PCIe and CXL protocols to carry various commands or events, such as link speed changes, reset commands issued by the host, and other reliability, availability, and serviceability features.

[0043] In an example, CXL link bus 206 (eg, corresponding to Figure 1104) may support communications using multiplexed protocols for caching (e.g., CXL.cache), memory access (e.g., CXL.mem or CXL.memory), and data input / output transactions (e.g., CXL.io). CXL.io may include a PCIe-based protocol for functions such as device discovery, configuration, initialization, I / O virtualization, and direct memory access (DMA) using non-uniform load-store, producer-consumer semantics. CXL.cache may enable a device to cache data from a host memory (e.g., from host memory 214) using a request and response protocol. CXL.memory may enable a host device 202 to use memory attached to a CXL device 204, for example, in or using a virtualized memory space. CXL-based memory devices may include or use volatile or non-volatile memory, such that they may be characterized by different speeds or latencies. In an example, a CXL-based memory device may include a CXL-based memory controller configured to manage transactions with volatile or non-volatile memory.

[0044] In an example, CXL.memory transactions may be memory load and store operations run downstream from or external to the host device 202. CXL memory devices may have varying levels of complexity. For example, a simple CXL memory system may include a CXL device that includes or is coupled to a single media controller, such as a memory controller (MEMC). A medium CXL memory system may include a CXL device that includes or is coupled to multiple media controllers. A complex CXL memory system may include a CXL device that includes or is coupled to a cache controller (and its attached caches) and to one or more media or memory controllers.

[0045] exist Figure 2 In an example of , host device 202 includes a host processor 216 (e.g., including one or more CPUs or cores) and an IO device 228. Host device 202 may include or may be coupled to host memory 214. Host device 202 may include various circuitry or logic configured to facilitate CXL-based communications and transactions with CXL device 204. For example, host device 202 may include coherence and memory logic 220 configured to implement transactions according to CXL.cache and CXL.memory semantics, and host device 202 may include PCIe logic 222 configured to implement transactions according to CXL.io semantics. In an example, host device 202 may be configured to manage the coherence of data cached at CXL device 204 using, for example, its coherence and memory logic 220.

[0046] Host device 202 may further include a host multiplexer 218 configured to modulate communications on CXL link bus 206 (e.g., using a PCIe PHY layer). Multiplexing of protocols ensures that latency-sensitive protocols (e.g., CXL.cache and CXL.memory) have the same or similar latency as a local processor-to-processor link. In an example, CXL defines an upper bound on the response time of latency-sensitive protocols to help ensure that device performance is not adversely affected by latency variations between different devices implementing coherency and memory semantics.

[0047] In an example, a symmetric cache coherence protocol may be difficult to implement between host processors because different architectures may use different solutions, which in turn may compromise backward compatibility. CXL may address this problem by merging coherence functions at host device 202 (eg, using coherence and memory logic 220).

[0048] The CXL device 204 may include various components or logic blocks, including a CXL host interface 232 and a device management system 234. In an example, the CXL host interface 232 may be configured to receive and manage various requests and transactions. For example, the CXL host interface 232 may be configured to receive and communicate PCIe resets, such as using PERST (PCI Express Reset), Warm Reset, FLR (Function Level Reset), and CXL Reset. In an example, the CXL host interface 232 may be configured to receive and communicate DOE transaction layer packets. In an example, the CXL host interface 232 may be configured to handle sideband requests or other miscellaneous events from PCIe and CXL devices, such as using the CXL link bus 206 or the system management bus 208.

[0049] The CXL host interface 232 may include or use a plurality of CXL interface physical layers 212. The device management system 234 may include a device logic and memory controller 224, among other things. In an example, the CXL device 204 may include a device memory 230, or may be coupled to another memory device. The CXL device 204 may include various circuitry or logic configured to facilitate CXL-based communications and transactions with the host device 202 using a CXL link bus 206. For example, the device logic and memory controller 224 may be configured to implement transactions received using the CXL host interface 232 according to CXL.cache, CXL.memory, and CXL.io semantics. The CXL device 204 may include a CXL device multiplexer 226 configured to control communications on the CXL link bus 206.

[0050] In an example, one or more of the coherence and memory logic 220, the device management system 234, and the device logic and memory controller 224 include a unified assistance engine (UAE) or a computing fabric with various functional units, such as a command manager (CM), a thread engine (TE), a stream engine (SE), a data manager or a data mover (DM), or other units. The computing fabric is reconfigurable and can include separate synchronous and asynchronous streams.

[0051] Device management system 234 or device logic and memory controller 224 or portions thereof may be configured to operate in the application space of CXL system 200 and in some instances may launch its own threads or sub-threads which may operate in parallel and may optionally use resources or units on other CXL devices 204. Queue and transaction control through the system may be coordinated by the CM, TE, SE or DM components of the UAE. In an example, each queue or thread may be mapped to a different loop iteration to thereby support multi-dimensional loops. With the ability to launch such nested loops, among other capabilities, the system may achieve significant time savings and latency improvements for compute-intensive operations.

[0052] In an example, command fences may be used to help maintain order throughout such operations, which may be performed locally or throughout the computational space of device logic and memory controller 224. In some examples, a CM may be used to route commands to a specific command execution unit (e.g., device logic and memory controller 224 including a specific instance of CXL device 204) using an out-of-order interconnect that provides corresponding transaction identifiers (TIDs) to command and response message pairs.

[0053] In an example, the CM can coordinate synchronization flows, such as using an asynchronous fabric of a reconfigurable computing fabric to communicate with other synchronization flows and / or other components of the reconfigurable computing fabric using asynchronous messages. For example, the CM can receive an asynchronous message from a dispatch interface and / or from another flow controller indicating a new thread at or using a synchronization flow. The dispatch interface can interface between the reconfigurable computing fabric and other system components. In some examples, a synchronization flow can send an asynchronous message to the dispatch interface to indicate thread completion.

[0054] Asynchronous messages may be used by synchronous streams, for example, to access memory. For example, a reconfigurable computing fabric may include one or more memory interfaces. A memory interface is a hardware component that may be used by a synchronous stream or its components to access external memory that is not part of the synchronous stream but is accessible by the host device 202 or the CXL device 204. Threads executed using synchronous streams may include sending read and / or write requests to the memory interface. Because reads and writes are asynchronous, a thread that initiates a read or write request to the memory interface may not receive the result of the request. Instead, the result of the read or write request may be provided to a different thread executing at a different synchronous stream. Delay and output registers in one or more of the CXL devices 204 may help coordinate and maximize the efficiency of a first stream, for example, by precisely timing the engagement of specific computing resources of one device with the arrival of data related to the first stream. For example, when a first stream resides or waits for other data or operations to complete, registers may help enable specific computing resources of the same resource to be reused for streams other than the first stream. Such other data or operations may depend on one or more other resources of the fabric.

[0055] Figure 3 An example of a portion of a CXL system that may include or use a virtual hierarchy for managing transactions (eg, memory transactions with CXL memory devices) is generally shown. The example may include or use real-time telemetry to help facilitate allocation of new or ongoing queues. Figure 3 Instances of include first virtual layer 304 and second virtual layer 306. First virtual layer 304, second virtual layer 306, or one or more modules or components thereof may be implemented using host device 202, CXL device 204, or multiple instances of host device 202 or CXL device 204.

[0056] exist Figure 3 In the example of FIG. 3 , the first virtual layer 304 includes a first host device 308 and the second virtual layer 306 includes a second host device 310. A CXL switch 302 may be provided to expose multiple CXL resources to different hosts in the system. In other words, the CXL switch 302 may be configured to couple each of the first host device 308 and the second host device 310 to the same or different resources, for example, using respective virtual CXL switches (VCSs), such as the first VCS 320 and the second VCS 322, respectively. The CXL switch 302 may be statically configured to couple each host device to respective different resources or the CXL switch 302 may be dynamically configured to different resources, for example, depending on the needs of a particular one of the host devices to execute its respective queue or thread. Thus, the CXL switch 302 enables virtual layers and resource sharing among different hosts.

[0057] In an example, a fabric manager (FM) may be provided to assign or coordinate connectivity of CXL switches 302 and may be configured to initiate, decompose or reconfigure virtual hierarchies of a CXL system. The FM may include a baseboard management controller (BMC), an external controller, a centralized controller or other controller.

[0058] exist Figure 3 In the example of , the CXL switch 302 or the first VCS 320 or the second VCS 322 may coordinate the communication between the host device and the various accelerators. For example, the CXL switch 302 may be coupled to various CXL devices (e.g., the first CXL device 318 or the second CXL device 324) via multiple logical devices (MLDs, such as the MLD 312) or to various logical devices, such as a single logical device (LD, such as the first LD 314, the second LD 316, the third LD 326, or the fourth LD 328). Each CXL device and logical device may represent a corresponding accelerator or CXL device with its own corresponding CXL.io configuration space, CXL.mem memory space, or CXL.cache cache space.

[0059] Figure 4 An example of an example CXL device 402 is generally shown configured to connect to a host device, such as host device 202, or to one or more other CXL devices using CXL link bus 206 and system management bus 208. Example CXL device 402 may include an example of CXL device 204, first CXL device 318, second CXL device 324, or another device.

[0060] For ease of illustration and discussion, the example CXL device 402 may include a conceptual front end portion, a middle end portion, and a back end portion. The portions and their components may be configured or combined differently according to different implementations of the example CXL device 402. Figure 4 In the example of the embodiment of the present invention, the front end portion may include a CXL link bus 206, a system management bus 208, a CXL host interface 404, a CXL interface physical layer 406, and an I / O path hardware logic 408. In the example, the front end portion includes a CXL data link layer and a CXL transport layer configured to manage various transactions with the host. In the example, the CXL transport layer includes registers and operators configured to manage a CXL request queue (e.g., including one or more memory transaction requests) and a CXL response queue (e.g., including one or more memory transaction responses) of the example CXL device 402.

[0061] The example CXL device 402 may include or use a CXL host interface 404, e.g., including one or more CXL interface physical layers 406 and I / O path hardware logic 408, coupled to the CXL link bus 206 or the system management bus 208, and using a multiplexing protocol for cache (e.g., CXL.cache), memory access (e.g., CXL.mem or CXL.memory), and data input / output transactions (e.g., CXL.io). The CXL host interface 404 may include the CXL host interface described above in Figure 2 An example of CXL host interface 232 is described in the example of FIG.

[0062] Depending on the operating mode of the example CXL device 402, data transactions may be passed from the I / O path hardware logic 408 to one or both of the host interface circuit 410 and the subsystem manager circuit 412. In an example, the CXL host interface 404 and the I / O path hardware logic 408 include normally-on logic of the example CXL device 402 that is not reset during internal reset operations involving one or both of the host interface circuit 410 and the subsystem manager circuit 412. In an example, the I / O path hardware logic 408 may be configured to interface directly with the memory controller 414.

[0063] The example CXL device 402 may be a memory device that includes a cache (e.g., including SRAM) and may include longer term volatile or non-volatile memory accessible via a memory controller. For example, the example CXL device 402 may include a cache memory in a middle end portion of the device. The middle end portion may further include a cache controller configured to monitor requests from the CXL transport layer and identify requests that may be fulfilled using the cache memory.

[0064] Various complications may arise in a CXL system. For example, CXL transactions may be based on a relatively large transaction size (e.g., 64 bytes), while some processes may use finer granularity or smaller data sizes. Thus, in some examples, a cache controller may be included or used in the device to store excess data fetched from a backend media controller or memory (e.g., from one or more memories in the backend portion of example CXL device 402).

[0065] In an example, the cache controller is coupled to a forked switch interface or an XBAR interface. The XBAR interface can be configured to allow multiple requestors to access multiple memory controllers in parallel, such as multiple memory controllers in a back end portion of a device. In an example, the XBAR interface provides essentially point-to-point access between requestors and memory controllers and provides performance that is generally higher than that available using conventional bus architectures. The XBAR interface can be configured to receive a response from the back end portion or receive a cache hit from the cache memory and deliver the response to the front end portion using a cache response queue.

[0066] In an example, the backend portion of the example CXL device 402 includes one or more memory controllers, including a first memory controller 414. Each of the memory controllers may have or use a respective memory request and response queue. Each of the memory controllers may be coupled to a respective medium or memory, such as may include volatile or non-volatile memory. In an example, each of the multiple memory controllers in the system may manage its own respective queue.

[0067] In an example, the middle end portion of the example CXL device 402 may include a host interface circuit 410 and a subsystem manager circuit 412. The host interface circuit 410 may be configured to manage some or all communications or transactions with a host device, may be configured to manage host-initiated device resets, and may be configured to manage sideband communications, such as using the system management bus 208. In an example, the subsystem manager circuit 412 is configured to manage CXL device specific behaviors or operations. For example, the example CXL device 402 may be a type 1, type 2, or type 3 CXL device, etc. In an example, the example CXL device 402 may include a memory device and the subsystem manager circuit 412 may include a memory controller 414 or may include circuitry for interfacing with the memory controller 414.

[0068] The example CXL device 402 may further include or use one or more device-specific controllers or logic. For example, the example CXL device 402 may include a CXL-based memory device having a memory controller 414. The memory controller 414 may be configured to control a device memory 416, such as a volatile or non-volatile memory. In a specific example, the device memory 416 includes DRAM or SRAM memory devices, or a combination of DRAM and SRAM memory devices, and transactions with such devices are managed by the memory controller 414 under the control or direction of the subsystem manager circuit 412.

[0069] In an example, the host interface circuit 410 and the subsystem manager circuit 412 may be configured to work together to help manage reset operations of the example CXL device 402. A reset may be requested by a host device or may be initiated internally to the example CXL device 402, such as to perform a firmware update or to recover from a serious failure (e.g., by restarting). The reset operation may include, for example, pulling down a device port, powering off a CPU or other logic on the device, or otherwise resetting or restarting one or all portions of the example CXL device 402, which in turn may result in loss of information on an in-band channel (i.e., a portion of the CXL link bus 206 that carries CXL transactions) or a sideband channel (e.g., using an I2C or I3C protocol) of the system management bus 208. In examples involving a device-initiated reset, the host device may not be notified of the reset, or the example CXL device 402 may not have sufficient time or capacity to issue a reset notification to the host. Without a priori knowledge of a reset at the example CXL device 402, the host device may expect that transactions or other communications on the CXL link bus 206 or the system management bus 208 will not be disturbed. In an example, the example CXL device 402 may use any of the host interface circuit 410 and the subsystem manager circuit 412 to help maintain communications with the host device during a reset of the example CXL device 402 or a portion thereof. That is, at least a portion of the example CXL device 402 may be configured to operate and maintain communications with the host device such that a connection between the host device and the example CXL device 402 is not lost when all or a portion of the example CXL device 402 undergoes a reset.

[0070] In an example, firmware updates of an example CXL device 402 may be used to add or change device features or operations. The CXL specification provides mailbox commands to update firmware. The sideband protocol may also have a defined mechanism for updating firmware. To update firmware, firmware instructions or updates may be loaded at the example CXL device 402 and then an internal reset (i.e., inside the example CXL device 402) may be performed so that the device CPU may begin running the new or updated firmware.

[0071] In an example, a failover or emergency recovery may include a reset initiated in response to an error condition detected at and optionally by the example CXL device 402. In an example, a failover reset may include collecting a memory snapshot for debugging and after collecting the data, performing a reset or restart to determine whether the device can be recovered. The failover reset may be a device-level reset and not in response to or based on a command from a host device.

[0072] The inventors have recognized that performing a reset at the example CXL device 402 may include or use in conjunction with a plurality of logic circuits or processors to help ensure that communications with the host device are maintained and host commands are acknowledged throughout the reset operation. A solution may include or use a host interface circuit 410 and a subsystem manager circuit 412 so that a portion of the example CXL device 402 has at least one logic block that is always on and available to handle communications on the CXL link bus 206 or the system management bus 208.

[0073] Again, refer to Figure 2 , CXL device 204 includes a centralized device management system 234. Thus, to reset CXL device 204, device management system 234 is shut down and reset, and the connection with the host device may be lost. In contrast, Figure 4 The example CXL device 402 includes a host interface circuit 410 and a subsystem manager circuit 412, each configured to manage a host connection throughout a reset.

[0074] For example, the host interface circuit 410 may be configured to maintain a connection to the host device while quiescing an IO path of the CXL link bus 206. For example, the host interface circuit 410 may throttle communications by applying back pressure on the CXL link bus 206, such as by issuing less or zero credits to the host device to slow or stop traffic on the CXL link bus 206 or the system management bus 208 between the host device and the example CXL device 402. In various examples, throttling communications may include reducing the number or volume of communications without completely stopping or eliminating communications using a particular channel or a particular bus or link. In other examples, throttling communications may include stopping or eliminating communications along a channel, bus, or link. In an example, throttling communications to reset an IO path includes pausing or stopping memory device commands or packets while allowing other packets to flow to maintain a CXL connection between the host device and the CXL device 402. While the host interface circuit 410 maintains a connection with the host device, all or a portion of the subsystem manager circuit 412, or circuits under its control such as the memory controller 414 or the device memory 416, etc., may be reset. The reset may be customizable or configurable, and may involve the reset of one or more functional blocks of the subsystem manager circuit 412. When the subsystem manager circuit 412 reset is complete, the host interface circuit 410 may then relax the back pressure and begin issuing additional credits to the host to resume normal traffic and command handling.

[0075] In another example, the subsystem manager circuit 412 may be configured to maintain a connection to the host device when the host interface circuit 410 or a portion thereof undergoes a reset. That is, the subsystem manager circuit 412 may be configured to apply back pressure on the CXL link bus 206 by managing credits issued from the example CXL device 402 to the host device. When the reset of the host interface circuit 410 is complete, the host interface circuit 410 and the subsystem manager circuit 412 may then be used together to resume normal operation.

[0076] CXL device operation mode

[0077] In an example, the subsystem manager circuit 412 may be configured to operate in a variety of modes including an "autonomous mode" and a "distributed mode". Normal or default operation may use the distributed mode. In the distributed mode, no device internal reset occurs, and all hardware and firmware of the CXL device may be considered operational. When a reset is required, the subsystem manager circuit 412 may direct the host interface circuit 410 to use the autonomous mode. In the autonomous mode, a device internal reset may occur. In the autonomous mode, the host interface circuit 410 may operate while the subsystem manager circuit 412 or a portion thereof is undergoing a reset and is therefore unavailable for managing transactions on the CXL link bus 206 or the system management bus 208. That is, the host interface circuit 410 may be configured to handle sideband and mainband transactions on the system management bus 208 and the CXL link bus 206, respectively, using, for example, the CXL host interface 404. In an example, the subsystem manager circuit 412 may be configured to determine when or whether to change the operating mode between the distributed mode and the autonomous mode.

[0078] The CXL host interface 404 may be configured to manage requests, for example, for PCIe resets, CXL resets, DOE transaction layer packets, and sideband commands (e.g., regarding link speed changes), as similarly described above for the CXL host interface 232 of the CXL device 204. Such requests may be handled by the example CXL device 402 in autonomous mode and in distributed mode. However, in autonomous mode, the subsystem manager circuitry 412 may be stalled or unavailable, and accordingly the CXL host interface 404 may operate in conjunction with or under the direction of the host interface circuitry 410 to handle such requests.

[0079] In an example, when the host interface circuit 410 is directed to use autonomous mode (e.g., by the subsystem manager circuit 412), the host interface circuit 410 may assume that the CXL.mem path is clean because the subsystem manager circuit 412 has or begins to apply back pressure to the bus to reduce the number of new requests or transactions arriving at the instance CXL device 402. In an example, the transition to autonomous mode may include checking whether device attached memory is initialized, and after a specified delay, the device cache may be unloaded (e.g., to the device memory 416 or elsewhere) to ensure that any cache managed by the instance CXL device 402 is clean, such as before a reset.

[0080] In an example, subsystem manager circuitry 412 may include logic for detecting when autonomous mode is used. For example, when example CXL device 402 completes a cold boot sequence and before the host enables device memory, logic may initiate autonomous mode operation. In this case, because device memory has not yet been enabled, there is no traffic and therefore the entire memory subsystem does not need to be managed. Therefore, host interface circuitry 412 may operate alone, and one or more portions of subsystem manager circuitry 412 may be powered off or quiesced.

[0081] In a second example, when a host-initiated PCIe or CXL reset is complete, the hardware or software implemented logic of the subsystem manager circuitry 412 may initiate autonomous mode operation. In this example, the host is aware of the reset operation and the device may wait for the host to enable memory or other portions of the example CXL device 402. Thus, the host interface circuitry 410 may operate alone, and one or more portions of the subsystem manager circuitry 412 may remain powered off or quiescent.

[0082] In a third example, when the CXL device initiates an internal reset, the logic of the subsystem manager circuitry 412 may initiate autonomous mode operation. In this case, the device firmware may quiesce the device port and clean up pending commands (e.g., by waiting or aborting), and may initialize device memory. In the case of a reset due to a device failure or freeze, cleanup may be expedited and performed without delay to ensure completion of the maximum number of pending commands. In the case of a reset due to a firmware update, cleanup may be performed while back pressure is applied to the CXL link bus 206. After cleanup, including, for example, moving cached data to the device memory 416, the host interface circuitry 410 may operate alone while the subsystem manager circuitry 412 is reset.

[0083] Thus, the subsystem manager circuitry 412 may be configured to detect various conditions or circumstances that indicate a need to operate the example CXL device 402 in an autonomous mode, which may include, for example, instructing the host interface circuitry 410 to temporarily take over management of transactions with the host device when the subsystem manager circuitry 412 is reset or otherwise unavailable. In general, among other circumstances, the example CXL device 402 may operate in a distributed mode, in which the host interface circuitry 412 manages main-band and / or side-band transactions with the host and the subsystem manager circuitry 412 controls device internal processes in response to host commands.

[0084] Figure 5 An example of a first timing diagram 500 is generally shown for interface management during, for example, an internal reset of an example CXL device 402. In the example, the first timing diagram 500 generally shows a process that may include transitioning portions of the example CXL device 402 from a distributed mode to an autonomous mode, and updating firmware of the subsystem manager circuit 412.

[0085] The first timing diagram 500 may begin at operation 502, where a mailbox command for a device reset is received via CXL link bus 206. For example, the command may be sent by a host device. In an example, operation 502 includes receiving a mailbox command that causes an internal reset to complete command processing. In an example, at operation 504, subsystem manager circuit 412 receives the mailbox command and initiates an acknowledgement or other device response.

[0086] At operation 506, the subsystem manager circuit 412 may command the subsystems of the instance CXL device 402 to quiesce or pause activity. For example, operation 506 may include managing pending mainband or sideband commands, shutting down interfaces or other ports, and reducing command counts to zero. Operation 506 may include flushing queues, hardware FIFOs, caches, or other components that are to be cleared at reset. In an example, operation 506 includes applying back pressure to commands on the CXL link bus 206 or system management bus 208, or issuing few or zero credits to the host device to ensure that few or zero commands arrive at the instance CXL device 402 during reset.

[0087] At operation 508, the subsystem manager circuit 412 may notify the host interface circuit 410 that the subsystem manager circuit 412 will begin operating in the autonomous mode and that the host interface circuit 410 may undergo a reset. In an example, at operation 510, the subsystem manager circuit 412 may store reset context information, such as a device state, a register state, or other information about the device at or before the time the command was received at operation 502. After the context load at operation 510, operation 512 may include using the subsystem manager circuit 412 to instruct the host interface circuit 410 to perform a reset. In response, the host interface circuit 410 may suspend the process at operation 514, and after a specified delay or after the command count or queue reaches zero, operation 516 may include resetting the host interface circuit 410.

[0088] When the host interface circuit 410 reset is complete, the host interface circuit 410 may be initialized to manage communications with the host device. Thus, at operation 518, the subsystem manager circuit 412 may begin its reset. For example, operation 518 may include or use ROM code (e.g., newly loaded or updated code) to initialize the subsystem manager circuit 412. While the subsystem manager circuit 412 is loading, the host interface circuit 410 may handle any pending events and maintain a connection with the host device at operation 520. Because the host interface circuit 410 is already operating and managing the host connection, at operation 522, initialization of the host interface circuit 410 (e.g., including a power cycle) may be skipped. At operation 524, a boot loader may be used to load relevant code or applications into the registers of the subsystem manager circuit 412 and the subsystem manager circuit 412 may resume normal operation. In an example, at operation 524, the boot loader starts and loads a firmware image (e.g., a new or updated firmware image) to the subsystem manager circuit 412, and then loads the firmware image (e.g., a new or updated firmware image) to the host interface circuit 410. The host interface circuit 410 is suspended or in a paused state while the host interface circuit 410 firmware image is loaded into its IMEM. If any host events occur during the firmware image loading, a timeout error may occur. To help avoid this, the host interface events may be unmasked in the subsystem manager circuit 412 so that the subsystem manager circuit 412 can handle such host events and avoid timeouts.

[0089] Figure 6An example of a second timing diagram 600 is generally shown for interface management during, for example, an internal reset of the example CXL device 402. In the example, the second timing diagram 600 is a continuation of the first timing diagram 500 and begins at operation 602 after operation 524. In the example, the second timing diagram 600 generally illustrates a process that may include updating firmware of the host interface circuit 410, and returning operation of the example CXL device 402 to a distribution mode.

[0090] In the example of the second timing diagram 600, operation 602 may include or use the subsystem manager circuit 412 to mask the host interface events. At operation 604, the second timing diagram 600 may unmask the host interface events. In the example, operation 602 includes masking the host events on the host interface circuit 410, and at operation 604, unmasking the host events on the subsystem manager circuit 412. In this example, the boot loader handles the host events in the subsystem manager circuit 412.

[0091] At operation 606, new firmware may be loaded to the host interface circuit 410. In an example, the subsystem manager circuit 412 may manage the loading and implementation of the new firmware for the host interface circuit 410. Operation 606 may include loading (e.g., using a boot loader) the new firmware and initializing the host interface circuit 410. At operation 608, the subsystem manager circuit 412 may command the host interface circuit 410 to operate in an autonomous mode until additional instructions are sent to the host interface circuit 410 by the subsystem manager circuit 412.

[0092] At operation 610, subsystem manager circuitry 412 may mask host interface events for instance CXL device 402, and at operation 612, subsystem manager circuitry 412 may unmask host interface events. While interface events are masked or handled by subsystem manager circuitry 412, host interface circuitry 410 may remain operating in autonomous mode (operation 614).

[0093] At operation 616, the second timing diagram 600 may include waiting for a host command, such as a memory enable command from the host device. Upon receiving such a command, the subsystem manager circuit 412 may initiate a distribution mode of the example CXL device 402. At operation 620, the host interface circuit 410 and the subsystem manager circuit 412 may operate in the distribution mode.

[0094] Figure 7An example of a first method 700 for resetting at least a portion of a CXL device is shown. The first method 700 depicts a specific sequence of operations that may be altered without departing from the scope of the present disclosure. For example, some of the operations depicted may be performed in parallel or in a different sequence. In other examples, different components of an example device or system implementing the first method 700 may perform functions at substantially the same time or in a specific sequence.

[0095] In an example, the first method 700 includes detecting a reset condition at a CXL device at operation 702. For example, operation 702 may include detecting a reset condition or receiving a reset trigger at the example CXL device 402. The reset condition or trigger may include or may indicate a firmware update, a detected device error or failure, or other conditions. In an example, the reset condition or trigger may include a host-initiated reset command. In an example including an example CXL device 402 operable in multiple modes, including an autonomous mode and a distributed mode, operation 702 may include initiating an autonomous mode.

[0096] At operation 704, the first method 700 may include quiescing transactions between the CXL device and the host device. In an example, operation 704 includes reducing the number of CXL transaction credits issued by the example CXL device 402 to the host device to zero.

[0097] At operation 706, the first method 700 may include maintaining a connection between the example CXL device 402 and the host device using the host interface circuit 410 of the example CXL device 402. Maintaining the connection at operation 706 may include acknowledging a mainband or sideband request or command from the host device.

[0098] At operation 708, the first method 700 may include resetting the subsystem manager circuit 412 of the example CXL device 402. Resetting the subsystem manager circuit 412 may include, for example, loading new or updated ROM code, or changing other configuration registers or commands of the example CXL device 402, and may further include restarting and initializing the subsystem manager circuit 412.

[0099] At operation 710, the first method 700 may include maintaining a connection between the instance CXL device 402 and a host device using the subsystem manager circuit 412 of the instance CXL device 402. Maintaining the connection at operation 710 may include acknowledging a mainband or sideband request or command from the host device.

[0100] At operation 712, the first method 700 may include resetting the host interface circuit 410 of the example CXL device 402. Resetting the host interface circuit 410 may include, for example, loading new or updated ROM code, or changing other configuration registers or commands of the example CXL device 402, and may further include restarting and initializing the host interface circuit 410.

[0101] After operation 712, any new or updated firmware for the example CXL device 402 may be installed and all portions of the example CXL device 402 may be ready for use. However, in some examples, the host interface circuit 410 may remain uninitialized or otherwise partially or completely powered off until full operation of the host interface circuit 410, the subsystem manager circuit 412, or the example CXL device 402 is requested by the host device. For example, at operation 714, the first method 700 may include receiving a command from the host device to enable or use a memory device coupled to or controlled by the example CXL device 402. For example, the memory device may be controlled by the subsystem manager circuit 412 of the example CXL device 402. In response to receiving the host command, the host interface circuit 410 may be initialized and started. Then, the host interface circuit 410 may resume managing host commands, host resets, and / or sideband communications, while the subsystem manager circuit 412 may be used to initialize the memory device and perform other operations or functions of the example CXL device 402. Thus, operation 716 may include or use host interface circuitry 410 and subsystem manager circuitry 412 to manage subsequent transactions with the host device.

[0102] Figure 8A block diagram of an example machine 800 is shown in which any one or more of the techniques (e.g., methodology) discussed herein may be implemented, implemented in, or implemented by. As described herein, the example may include or be operated by logic or several components or mechanisms in the machine 800. A circuit system (e.g., a processing circuit system) is a collection of circuits implemented in a tangible entity of the machine 800 that includes hardware (e.g., simple circuits, gates, logic, etc.). Over time, the circuit system membership (e.g., belonging to a host-side device or process, or belonging to an accelerator-side device or process) may change flexibly. The circuit system includes members that can perform specified operations individually or in combination when operating. In an example, the hardware of the circuit system may be designed invariably to implement a specific operation (e.g., hardwired), such as using device logic and memory controller 224, or host interface circuit 410, or subsystem manager circuit 412, or using its specific command execution unit. In an example, the hardware of the circuit system may include variably connected physical components (e.g., command execution units, transistors, simple circuits, etc.) that include machine-readable media that are physically modified (e.g., magnetic, electrical, removable placement of constant mass particles, etc.) to encode instructions for specific operations. When the physical components are connected, the basic electrical properties of the hardware composition change from insulators to conductors, or vice versa, for example. The instructions enable embedded hardware (e.g., execution units or loading mechanisms) to create members of the circuit system in hardware via variable connections to implement portions of specific operations in operation. Therefore, in an example, the machine-readable media element is part of the circuit system or is communicatively coupled to other components of the circuit system when the device is operating. In an example, any of the physical components can be used in more than one member of more than one circuit system. For example, when operating, an execution unit can be used in a first circuit of a first circuit system at one point in time and reused by a second circuit in the first circuit system or by a third circuit in the second circuit system at a different time.

[0103] In alternative embodiments, the machine 800 may operate as a standalone device or may be connected (e.g., networked) to other machines. In a networked deployment, the machine 800 may operate as a server machine, a client machine, or both in a server-client network environment. In an example, the machine 800 may act as a peer machine in a peer-to-peer (P2P) (or other distributed) network environment. The machine 800 may be a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile phone, a network appliance, a network router, a switch or a bridge, or any machine capable of executing (sequentially or otherwise) instructions specifying actions to be taken by the machine. Furthermore, although only a single machine is shown, the term "machine" shall also be construed to include any collection of machines that individually or jointly execute one (or more) sets of instructions to perform any one or more of the methodologies discussed herein (e.g., cloud computing, software as a service (SaaS), other computer cluster configurations).

[0104] Any one or more of the components of machine 800 may include or use one or more instances of host device 202 or CXL device 204 or instance CXL device 402 or other components in or attached to computing system 100. Machine 800 (e.g., a computer system) may include hardware processor 802 (e.g., host processor 216, device logic and memory controller 224, central processing unit (CPU), graphics processing unit (GPU), hardware processor core, or any combination thereof), main memory 804, static memory 806 (e.g., memory or storage for firmware, microcode, basic input output (BIOS), unified extensible firmware interface (UEFI), etc.), and mass storage device 808 or a stack of memory dies, hard drive, tape drive, flash storage device, or other block device), some or all of which may communicate with each other via interconnect 830 (e.g., a bus). Machine 800 may further include display device 810, alphanumeric input device 812 (e.g., keyboard), and user interface (UI) navigation device 814 (e.g., mouse). In an example, the display device 810, the input device 812, and the UI navigation device 814 may be a touch screen display. The machine 800 may additionally include a mass storage device 808 (e.g., a drive unit), a signal generating device 818 (e.g., a speaker), a network interface device 820, and one or more sensors 816, such as a global positioning system (GPS) sensor, a compass, an accelerometer, or other sensors. The machine 800 may include an output controller 828, such as a serial (e.g., universal serial bus (USB), parallel or other wired or wireless (e.g., infrared (IR), near field communication (NFC), etc.) connection for communicating with or controlling one or more peripheral devices (e.g., printers, card readers, etc.).

[0105] The hardware processor 802, main memory 804, static memory 806, or a register of a mass storage device 808 may be or include a machine-readable medium 822 on which is stored one or more sets of data structures or instructions 824 (e.g., software) embodying or used by any one or more of the techniques or functions described herein. The instructions 824 may also reside, completely or at least partially, within any of the registers of the hardware processor 802, main memory 804, static memory 806, or a mass storage device 808 during execution thereof by the machine 800. In an example, one or any combination of the hardware processor 802, main memory 804, static memory 806, or a mass storage device 808 may constitute the machine-readable medium 822. Although the machine-readable medium 822 is illustrated as a single medium, the term "machine-readable medium" may include a single medium or multiple media (e.g., a centralized or distributed database, or associated caches and servers) configured to store one or more instructions 824.

[0106] The term "machine-readable medium" may include any medium capable of storing, encoding, or carrying instructions executed by the machine 800 and causing the machine 800 to perform any one or more of the techniques of the present disclosure, or capable of storing, encoding, or carrying data structures used by or associated with such instructions. Non-limiting machine-readable medium examples may include solid-state memory, optical media, magnetic media, and signals (e.g., radio frequency signals, other photon-based signals, sound signals, etc.). In an example, a non-transitory machine-readable medium includes a machine-readable medium having a plurality of particles with a constant (e.g., stationary) mass, and is therefore composed of matter. Thus, a non-transitory machine-readable medium is a machine-readable medium that does not include a transitory propagating signal. Specific examples of non-transitory machine-readable media may include: non-volatile memory, such as semiconductor memory devices (e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)) and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0107] In an example, information stored or otherwise provided on machine-readable medium 822 may represent instructions 824, such as instructions 824 themselves or a format from which instructions 824 may be derived. Such a format from which instructions 824 may be derived may include source code, encoded instructions (e.g., in compressed or encrypted form), packaged instructions (e.g., divided into multiple packages), or the like. Information representing instructions 824 in machine-readable medium 822 may be processed by processing circuitry into instructions that implement any of the operations discussed herein. For example, deriving instructions 824 from the information (e.g., processed by processing circuitry) may include: compiling (e.g., from source code, object code, etc.), interpreting, loading, organizing (e.g., dynamically or statically linking), encoding, decoding, encrypting, unencrypting, packaging, unpacking, or otherwise manipulating the information into instructions 824.

[0108] In an example, the export of instructions 824 may include assembling, compiling, or interpreting information (e.g., by processing circuitry) to create instructions 824 from some intermediate or pre-processed format provided by machine-readable media 822. Information, when provided in multiple parts, may be combined, unpacked, and modified to create instructions 824. For example, the information may be in multiple compressed source code packages (or object code or binary executable code, etc.) on one or several remote servers. The source code packages may be encrypted when transiting over the network and decrypted, uncompressed, assembled (e.g., linked) if necessary, and compiled or interpreted (e.g., into a library, a stand-alone executable file, etc.) at the local machine and executed by the local machine.

[0109] The instructions 824 may further be transmitted or received via the network interface device 820 using a transmission medium over the communication network 826 utilizing any of a number of transmission protocols (e.g., frame relay, Internet Protocol (IP), transmission control protocol (TCP), user datagram protocol (UDP), hypertext transfer protocol (HTTP), etc.). Example communication networks may include a local area network (LAN), a wide area network (WAN), a packet data network (e.g., the Internet), a mobile telephone network (e.g., a cellular network), a plain old telephone (POTS) network, and a wireless data network (e.g., a wireless network called The Institute of Electrical and Electronics Engineers (IEEE) 802.11 series of standards, known as 802.16 family of standards), IEEE 802.15.4 family of standards, peer-to-peer (P2P) networks, etc. In an example, the network interface device 820 may include one or more physical jacks (e.g., Ethernet, coaxial, or telephone jacks) or one or more antennas to connect to the network 826. In an example, the network interface device 820 may include multiple antennas to communicate wirelessly using at least one of single-input multiple-output (SIMO), multiple-input multiple-output (MIMO), or multiple-input single-output (MISO) technology. The term "transmission media" should be taken to include any intangible media capable of storing, encoding, or carrying instructions executed by the machine 800, and includes digital or analog communication signals or other intangible media that facilitate such software communication. Transmission media is machine-readable media.

[0110] To better illustrate the methods and apparatus described herein, such as may be used to facilitate resetting a CXL device while maintaining a host connection, a set of non-limiting example embodiments are described below as digital identification examples.

[0111] Example 1 is a memory device comprising: a host interface circuit configured to receive commands from a host device via a compute express link (CXL) interconnect; and a subsystem manager circuit configured to control transactions with the memory device; wherein the host interface circuit is configured to inhibit communication with the host device and maintain a connection with the host device when the subsystem manager circuit experiences a reset initiated internally by the memory device.

[0112] In Example 2, the subject matter of Example 1 optionally includes the resetting being responsive to a detected device failure of the memory device.

[0113] In Example 3, the subject matter of Examples 1-2 optionally includes wherein the resetting is responsive to a firmware update of the memory device.

[0114] In Example 4, the subject matter of Examples 1-3 optionally includes, in an autonomous mode of operation, the host interface circuitry is configured to manage CXL and sideband commands, PCIe resets, and / or CXL resets received from the host device, and in a distributed mode of operation, the subsystem manager circuitry is configured to manage CXL and sideband commands received from the host device. In an example, in the distributed mode, PCIe PHY resets, PHY programming, and link CXL.io resets may be handled by the host interface circuitry, while IO path resets and device backend resets may be handled by the subsystem manager circuitry.

[0115] In Example 5, the subject matter of Example 4 optionally includes, in the autonomous operating mode and in the distributed operating mode, the host interface circuitry being configured to coordinate device responses to any one or more of a PCIe reset, a CXL reset (e.g., as notified using CXL.io to write to a memory mapped register of the CXL device), a transaction layer packet, and a sideband request received from the host device.

[0116] In Example 6, the subject matter of Examples 4-5 optionally includes a cache memory, wherein in the autonomous operating mode, the subsystem manager circuit is configured to load contents of the cache memory to the memory device before initiating the reset.

[0117] In Example 7, the subject matter of Examples 4-6 optionally includes the memory device, wherein the memory device is a DRAM memory device.

[0118] In Example 8, the subject matter of Examples 4-7 optionally includes the subsystem manager circuitry configured to determine whether to initialize the memory device before using the autonomous operating mode.

[0119] In Example 9, the subject matter of Examples 4-8 optionally includes logic circuitry configured to initiate the autonomous operating mode (1) after a cold boot and before the memory device is enabled by the host device, and / or (2) in response to a reset command received from the host device, and / or (3) in response to a reset initiated by the memory device.

[0120] In Example 10, the subject matter of Examples 4-9 optionally includes that, in the autonomous operating mode, the host interface circuit is configured to inhibit communications with the host device by indicating that zero CXL transaction credits are available for the host device.

[0121] Example 11 is a method comprising: detecting a memory device reset condition at a memory device, wherein the memory device is coupled to a host device using a compute fast link (CXL) interconnect; in response to the reset condition: during a first stage, pausing transactions with the host device, using host interface circuitry of the memory device to maintain a connection with the host device, and resetting a subsystem manager circuitry of the memory device; during a second stage following the first stage, using the subsystem manager circuitry of the memory device to maintain the connection with the host device while resetting the host interface circuitry of the memory device; and during a third stage following the second stage, using the host interface circuitry of the memory device and the subsystem manager circuitry of the memory device to manage transactions with the host device.

[0122] In Example 12, the subject matter of Example 11 optionally includes detecting the memory device reset condition includes detecting a fault condition internal to the memory device.

[0123] In Example 13, the subject matter of Examples 11-12 optionally includes detecting the memory device reset condition includes receiving an indication that a firmware update is available for the memory device.

[0124] In Example 14, the subject matter of Example 13 optionally includes quiescing a transaction with the host device includes allowing a memory device internal command to complete before resetting the subsystem manager circuitry of the memory device.

[0125] In Example 15, the subject matter of Examples 11-14 optionally includes detecting the memory device reset condition includes receiving a reset command from the host device to reset at least a portion of the memory device.

[0126] In Example 16, the subject matter of Examples 11-15 optionally includes initiating the third phase in response to receiving a command from the host device to enable or use the memory device.

[0127] In Example 17, the subject matter of Examples 11-16 optionally includes the first phase ending when a reset of the subsystem manager circuit is complete.

[0128] Example 18 is a method for operating a peripheral device (e.g., including a memory device, an accelerator device, or other device) that is coupled to a host device using a compute express link (CXL) interconnect, the method comprising: operating a host interface circuit of the peripheral device in one of an autonomous mode and a distributed mode, wherein in the autonomous mode, the host interface circuit is configured to maintain a connection between the host device and the peripheral device and inhibit communication between the host device and the peripheral device when a subsystem manager circuit of the peripheral device is unavailable, and wherein in the distributed mode, the host interface circuit is configured to allow communication between the host device and the subsystem manager circuit of the peripheral device.

[0129] In Example 19, the subject matter of Example 18 optionally includes initiating the autonomous mode in response to a reset command received by the peripheral device; and wherein in the autonomous mode and in response to the reset command, the subsystem manager circuit is configured to execute a reset routine that includes loading new firmware for the subsystem manager circuit.

[0130] In Example 20, the subject matter of Examples 18-19 optionally includes initiating the autonomous mode in response to a fault condition detected at the peripheral device; and wherein in the autonomous mode, the subsystem manager circuit is configured to perform a reset routine including: resetting the subsystem manager circuit while the host interface circuit maintains the connection between the host device and the peripheral device in response to the detected fault condition; and resetting the host interface circuit while the subsystem manager circuit maintains the connection between the host device and the peripheral device.

[0131] Example 21 is at least one machine-readable medium comprising instructions that, when executed by processing circuitry, cause the processing circuitry to perform operations for implementing any of Examples 1-20.

[0132] Example 22 is an apparatus comprising means for implementing any of Examples 1-20.

[0133] Example 23 is a system for implementing any of Examples 1-20.

[0134] Example 24 is a method for implementing any of Examples 1-10.

[0135] Each of these non-limiting examples may stand alone or may be combined with one or more of the other examples in various permutations or combinations.

[0136] The above detailed description includes reference to the accompanying drawings that form a part of the detailed description. The drawings show by way of illustration specific embodiments in which the invention can be practiced. These embodiments are also referred to herein as "examples". Such examples may include elements other than those shown or described. However, the inventors also contemplate examples in which only those elements shown or described are provided. In addition, the inventors also contemplate examples (or one or more aspects thereof) using any combination or arrangement of those elements shown or described with respect to a specific example (or one or more aspects thereof) or with respect to other examples (or one or more aspects thereof) shown or described herein.

[0137] In this document, as is common in patent documents, the term "a / an" is used to include one or more than one, independent of any other examples or uses of "at least one" or "one or more". In this document, the term "or" is used to refer to a non-exclusive "or", so that "A or B" may include "A but not B", "B but not A" and "A and B", unless otherwise indicated. In the appended claims, the terms "including" and "in which" are used as the common English equivalents of the corresponding terms "including" and "wherein". Moreover, in the appended claims, the terms "including" and "including" are open-ended, that is, a system, device, article or process containing elements other than the elements listed after this term in the claim is still considered to fall within the scope of the claim. In addition, in the appended claims, the terms "first", "second" and "third" etc. are used only as markers, and it is not intended to impose numerical requirements on their objects.

[0138] The above description is intended to be illustrative rather than restrictive. For example, the above examples (or one or more aspects thereof) may be used in combination with each other. For example, a person of ordinary skill in the art may use other embodiments after reviewing the above description. The above description is submitted with the understanding that the above description will not be used to interpret or limit the scope or meaning of the claims. Moreover, in the above specific embodiments, various features may be grouped together to simplify the present disclosure. This should not be interpreted as wishing that unclaimed public features are necessary for any claim. Specifically, the subject matter of the invention may have non-all features of a specific disclosed embodiment. Therefore, the attached claims are hereby incorporated into the specific embodiments, wherein each claim itself is a separate embodiment, and it is expected that such embodiments can be combined with each other in various combinations or arrangements. The scope of the present invention should be determined with reference to the attached claims and the full range of equivalents authorized by this claim.

Claims

1. A memory device, comprising: a host interface circuit configured to receive commands from a host device via a compute express link (CXL) interconnect; and processing logic circuitry configured to control transactions with the memory device; Wherein the host interface circuit is configured to inhibit communications with the host device and maintain a connection with the host device when the processing logic circuitry undergoes a reset initiated internally by the memory device.

2. The memory device of claim 1, wherein the resetting is responsive to a detected device failure of the memory device.

3. The memory device of claim 1, wherein the resetting is responsive to a firmware update of the memory device.

4. The memory device of claim 1 , wherein in an autonomous operating mode, the host interface circuit is configured to manage CXL and sideband commands received from the host device; and Wherein in a distribution mode of operation, the processing logic circuitry is configured to manage CXL and sideband commands received from the host device.

5. The memory device of claim 4, wherein in the autonomous operating mode and in the distributed operating mode, the host interface circuitry is configured to coordinate device responses to any one or more of a PCIe reset, a CXL reset, a transaction layer packet, and a sideband request received from the host device.

6. The memory device of claim 4, further comprising a cache memory; Wherein in the autonomous operating mode, the processing logic circuitry is configured to load contents of the cache memory to the memory device prior to initiating the reset.

7. The memory device of claim 4, further comprising the memory device, wherein the memory device is a DRAM memory device.

8. The memory device of claim 4, wherein the processing logic circuitry is configured to determine to initialize the memory device prior to using the autonomous operating mode.

9. The memory device of claim 4, comprising logic circuitry configured to initiate the autonomous operating mode (1) after a cold boot and before the memory device is enabled by the host device, or (2) in response to a reset command received from the host device, or (3) In response to a reset initiated by the memory device.

10. The memory device of claim 4, wherein in the autonomous operating mode, the host interface circuit is configured to inhibit communications with the host device by indicating that zero CXL transaction credits are available to the host device.

11. A method comprising: detecting a reset condition at a memory device, wherein the memory device is coupled to a host device using a compute express link (CXL) interconnect; In response to the reset condition: During a first phase, quiescing transactions with the host device, maintaining a connection with the host device using host interface circuitry of the memory device, and resetting processing logic circuitry of the memory device; during a second phase subsequent to the first phase, using the processing logic circuitry of the memory device to maintain the connection with the host device while resetting the host interface circuitry of the memory device; and During a third phase following the second phase, transactions with the host device are managed using the host interface circuitry of the memory device and the processing logic circuitry of the memory device.

12. The method of claim 11, wherein detecting the memory device reset condition comprises detecting a fault condition internal to the memory device.

13. The method of claim 11, wherein detecting the memory device reset condition comprises receiving an indication that a firmware update is available for the memory device.

14. The method of claim 13, wherein stalling transactions with the host device comprises allowing memory device internal commands to complete before resetting the processing logic circuitry, wherein the processing logic circuitry comprises subsystem manager circuitry of the memory device.

15. The method of claim 11, wherein detecting the memory device reset condition comprises receiving a reset command from the host device to reset at least a portion of the memory device.

16. The method of claim 11, wherein the third phase is initiated in response to receiving a command from the host device to enable the memory device.

17. The method of claim 11, wherein the first phase ends when a reset of the processing logic circuitry is complete.

18. A method for operating a peripheral device coupled to a host device using a compute express link (CXL) interconnect, the method comprising: operating a host interface circuit of the peripheral device in one of an autonomous mode and a distributed mode, wherein in the autonomous mode, the host interface circuit is configured to maintain a connection between the host device and the peripheral device and inhibit communication between the host device and the peripheral device when processing logic circuitry of the peripheral device is unavailable, and Wherein in the distribution mode, the host interface circuit is configured to allow communication between the host device and the processing logic circuitry of the peripheral device.

19. The method of claim 18, wherein the autonomous mode is initiated in response to a reset command received by the peripheral device; and Wherein in the autonomous mode and in response to the reset command, the processing logic circuitry is configured to execute a reset routine including loading new firmware for the processing logic circuitry, and wherein the processing logic circuitry includes subsystem manager circuitry.

20. The method of claim 18, wherein the autonomous mode is initiated in response to a fault condition detected at the peripheral device; and Wherein in the autonomous mode, the processing logic circuit system is configured to perform a reset routine comprising: in response to the detected fault condition, resetting subsystem manager circuitry while the host interface circuitry maintains the connection between the host device and the peripheral device; and The host interface circuit is reset while the subsystem manager circuit maintains the connection between the host device and the peripheral device.

Images