Systems and methods for managing memory resources

Abstract

To provide a system and a method for managing a memory resource.SOLUTION: A system according to the present invention includes a first memory server including a cache coherent switch and a first memory module, a second memory server, and a server link switch connected to the first memory server and the second memory server. The first memory module is connected to the cache coherent switch, and the cache coherent switch is connected to the server link switch.SELECTED DRAWING: Figure 1A

Description

[Technical Field]

[0001] The present invention relates to computing systems, and more particularly to a system and method for managing memory resources in a system that includes one or more servers. [Background technology]

[0002] This Background section is intended to provide context only. No disclosure of any embodiment or concept in this section is admitted to be prior art.

[0003] Some server systems may include a collection of servers connected by a network protocol. Each server in such a system may include processing resources (e.g., a processor) and memory resources (e.g., system memory). In some cases, it is advantageous for the processing resources of one server to access the memory resources of another server, and to do so while minimizing the processing resources of either server.

[0004] Therefore, there is a need for an improved system and method for managing memory resources in a system that includes one or more servers. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2020-201957 Summary of the Invention [Problem to be solved by the invention]

[0006] The present invention has been made in view of the above-described prior art, and an object of the present invention is to provide an improved system and method for managing memory resources in a system including one or more servers. [Means for solving the problem]

[0007] In order to achieve the above object, the present invention provides a data storage and processing system that includes one or more servers and one or more memory servers connected by a server link switch. Each memory server includes one or more memory modules connected to the server link switch via a cache coherent switch. Each memory module includes a controller (e.g., a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC)) that provides enhanced functionality. These capabilities include allowing the server to interact with the memory module without accessing a processor, such as a central processing unit (CPU), (e.g., by performing remote direct memory access (RDMA)).

[0008] The system includes a first memory server including a cache coherent switch and a first memory module, a second memory server, and a server link switch connected to the first memory server and the second memory server, wherein the first memory module is connected to the cache coherent switch, and the cache coherent switch is connected to the server link switch.

[0009] The server link switch according to the present invention, which has been made to achieve the above object, can disable the power supply to the first memory module.

[0010] The server link switch disables power to the first memory module by instructing the cache coherent switch to disable power to the first memory module, and the cache coherent switch disables power to the first memory module when instructed by the server link switch to disable power to the first memory module.

[0011] In order to achieve the above object, the cache coherent switch according to the present invention performs deduplication within the first memory module.

[0012] The cache coherent switch compresses the data and stores the compressed data in the first memory module.

[0013] The server link switch queries the state of the first memory module.

[0014] The server link switch queries the status of the first memory module via an intelligent platform management interface (IPMI).

[0015] Querying the status includes querying a status selected from the group consisting of a power status, a network status, and an error check status.

[0016] The server link switch aggregates cache requests directed to the first memory server.

[0017] The system further includes a third memory module connected to the server link switch, the server link switch maintaining a consistency level selected from the group consisting of strict consistency, sequential consistency, causal consistency, and processor consistency between data stored in the first memory server and data stored in the third memory server.

[0018] The cache coherent switch monitors the fullness of a first region of memory and moves data from the first region of memory to a second region of memory, the first region of memory being in volatile memory and the second region of memory being in persistent memory.

[0019] The server link switch includes a PCIe switch.

[0020] The server link switch includes a CXL switch.

[0021] The server link switch includes a ToR CXL switch.

[0022] The server link switch transmits data from the second memory server to the first memory server and performs flow control on the data.

[0023] The system further includes a third memory server connected to the server link switch, the server link switch receiving a first packet from the second memory server, receiving a second packet from the third memory server, and transmitting the first packet and the second packet to the first memory server.

[0024] A method of performing remote direct memory access in a computing system including a first memory server, a first server, a second server, and a server link switch connected to the first memory server, the first server, and the second server, wherein the first memory server includes a cache coherent switch and a first memory module, the first server includes an embedded program processing circuit, and the second server includes an embedded program processing circuit, includes receiving a first packet from the first server by the server link switch, receiving a second packet from the second server by the server link switch, and transmitting the first packet and the second packet to the first memory server.

[0025] The method further includes compressing data by the cache coherent switch and storing the data in the first memory module.

[0026] The method further includes querying the status of the first memory server by the server link switch.

[0027] The system comprises a cache coherent switch, a first memory server including a first memory module, a second memory server, and a server link switching means connected to the first memory server and the second memory server, wherein the first memory module is connected to the cache coherent switch, and the cache coherent switch is connected to the server link switching means. [Effects of the Invention]

[0028] The present invention allows a server to interact with a memory module without accessing a processor by performing remote direct memory access (RDMA). [Brief explanation of the drawings]

[0029] [Figure 1A] 1 is a block diagram of a system for connecting memory resources to computing resources using cache coherent connections according to one embodiment of the present invention. [Figure 1B] 1 is a block diagram of a system using an expansion socket adapter to connect memory resources to computing resources using a cache coherent connection according to one embodiment of the present invention. [Figure 1C] 1 is a block diagram of a system for aggregating memory using an Ethernet top of rack (ToR) switch according to one embodiment of the present invention. [Figure 1D] 1 is a block diagram of a system for aggregating memory using an Ethernet ToR switch and an expansion socket adapter according to one embodiment of the present invention. [Figure 1E] 1 is a block diagram of a system for aggregating memory according to one embodiment of the present invention; [Figure 1F] 1 is a block diagram of a system for aggregating memory using an expansion socket adapter according to one embodiment of the present invention. [Figure 1G]A block diagram of a system for partitioning a server according to an embodiment of the present invention. [Figure 2A] A flowchart for explaining a method of executing a Remote Direct Memory Access (RDMA) transfer by bypassing a processing circuit for the embodiments shown in FIGS. 1A to 1G according to an embodiment of the present invention. [Figure 2B] A flowchart for explaining a method of executing an RDMA transfer with the involvement of a processing circuit for the embodiments shown in FIGS. 1A to 1D according to an embodiment of the present invention. [Figure 2C] A flowchart for explaining a method of executing an RDMA transfer via a Compute Express Link (CXL) switch for the embodiments shown in FIGS. 1E and 1F according to an embodiment of the present invention. [Figure 2D] A flowchart for explaining a method of executing an RDMA transfer via the CXL switch shown in FIG. 1G according to an embodiment of the present invention.

Embodiments for Carrying Out the Invention

[0030] Hereinafter, specific examples of embodiments for carrying out the present invention will be described in detail with reference to the drawings. Referring to the drawings, the following detailed description is intended to describe exemplary embodiments of a system and method for managing memory resources provided herein, and is not intended to show only the forms in which this specification can be configured or utilized. This description presents the features of this specification related to the exemplary embodiments. However, it should be understood that the same or equivalent functions and structures can be achieved by different embodiments intended to be included within the scope of this specification. As described elsewhere in this specification, like numbers are intended to indicate like elements or features.

[0031] Peripheral Component Interconnect express (PCIe) refers to a computer interface with relatively high and variable latency that can limit its usefulness when connecting to memory. CXL is an open industry standard for communication over PCIe 5.0 that provides fixed, relatively short packet sizes and, as a result, can provide relatively high bandwidth and relatively low, fixed latency. CXL can therefore support cache coherence, making it suitable for connecting to memory. CXL is also used in servers to provide connections between hosts and accelerators, memory devices, and network interface circuits (or "network interface controllers" or "network interface cards" (NICs)).

[0032] Cache coherence protocols such as CXL are used for heterogeneous processing in scalar, vector, and buffered memory systems. CXL utilizes channels, retimers, the system's PHY layer, the logical aspects of the interface, and the PCIe 5.0 protocol to provide a cache-coherent interface. The CXL transaction layer includes two multiplexed subprotocols that run simultaneously over a single link: CXL.io, CXL.cache, and CXL.memory. CXL.io includes I / O semantics similar to PCIe. CXL.cache includes cache semantics, and CXL.memory includes memory semantics; both cache and memory semantics are optional. Like PCIe, CXL supports (i) base widths divisible into x16, x8, and x4, (ii) data rates of 32 GT / s and 128b / 130b decomposable into 8 GT / s and 16 GT / s, (iii) 300 W (75 W with a x16 connector), and (iv) plug and play. To support plug and play, a PCIe or CXL device link starts training with PCIe Gen1, negotiates with CXL, and completes training for Gen1-5 before initiating a CXL transaction.

[0033] In one embodiment, as described in further detail below, in a system including multiple servers connected together by a network, the use of CXL connections to a collection or “pool” of memory (e.g., a quantity of memory including multiple memory cells connected together) provides various advantages. For example, in addition to providing packet / switching functionality for CXL packets, a CXL switch with additional functionality (referred to herein as an “enhanced CXL switch”) is used to connect the collection of memory to one or more central processing units (CPUs) (or “central processing circuits”) and one or more network interface circuits (with enhanced functionality). Such a configuration allows (i) the collection of memory to include various forms of memory with different characteristics, (ii) the enhanced CXL switch virtualizes the collection of memory and stores data with different characteristics (e.g., access frequency) in the appropriate type of memory, and (iii) the enhanced CXL switch can support remote direct memory access (RDMA) such that RDMA can be performed with little or no involvement of the server's processing circuitry. As used herein, "virtualizing" memory means performing memory address translation between processing circuitry and memory.

[0034] The CXL switch (i) supports memory and accelerator decomposition through a single level of switching, (ii) enables time-division multiplexing between domains based on demand, allows resources to be taken offline or online between domains, and (iii) supports downstream port virtualization. In one embodiment, CXL is used to implement aggregated memory, which enables one-to-many and many-to-one switching (e.g., (i) allows multiple loop ports to connect to one endpoint, (ii) allows one root port to connect to multiple endpoints, and (iii) allows multiple root ports to connect to multiple endpoints), using aggregated devices that are divided into multiple logical devices, each with an LD-ID (logical device identifier). In such an embodiment, a physical device is divided into multiple logical devices, each of which is recognized as a respective initiator. A device has one physical function (PF) and multiple (e.g., 16) separate logical devices. In one embodiment, the number of logical devices (e.g., the number of partitions) is limited (e.g., 16), and there is also one control partition (of the physical function used to control the device).

[0035] In one embodiment, a fabric manager (i) performs device discovery and virtual CXL software creation and (ii) is used to bind virtual ports to physical ports. Such a fabric manager operates over a connection via SMBus sideband. The fabric manager may be implemented in hardware, software, firmware, or a combination thereof, and may reside, for example, in a host, in one of the memory modules 135, in the enhanced CXL switch 130, or elsewhere in the network. The fabric manager can issue commands, including commands that occur over the sideband bus or PCIe tree.

[0036] Referring to FIG. 1A , in one embodiment, a server system includes multiple servers 105 connected together by a top-of-rack (ToR) Ethernet switch 110. This switch is illustrated using the Ethernet protocol, although other suitable network protocols may be used. Each server includes one or more processing circuits 115, each connected to (i) system memory 120 (e.g., double data rate (version 4) (DDR4) memory or any other suitable memory), (ii) one or more network interface circuits 125, and (iii) one or more CXL memory modules 135. Each of the processing circuits 115 is an embedded program processing circuit, such as a CPU (e.g., an x86 CPU), a graphics processing unit (GPU), or an ARM processor. In one embodiment, the network interface circuit 125 is implemented on one of the memory modules 135 (on the same semiconductor chip or in the same module), or the network interface circuit 125 is packaged separately from the memory modules 135.

[0037] As used herein, a "memory module" is a package (e.g., a package including a printed circuit board and components connected to the printed circuit board, or an enclosure including a printed circuit board) that includes one or more memory dies, each of which includes a plurality of memory cells. Each memory die, or each of a set of groups of memory dies, is in a package (e.g., an epoxy molding compound (EMC) package) soldered to the memory module's printed circuit board (or connected to the memory module's printed circuit board via a connector). Each of the memory modules 135 has a CXL interface and includes a controller 137 (e.g., an FPGA, ASIC, processor, and / or the like) for converting between CXL packets and the memory interface of the memory die, e.g., to signals appropriate for the memory technology of the memory within the memory module 135. As used herein, the "memory interface" of a memory die is an interface specific to the technology of the memory die, e.g., in the case of DRAM, the memory interface is word lines and bit lines. The memory module also includes a controller 137 that provides enhanced functionality, as described in more detail below. The controllers 137 of each of the memory modules 135 are connected to the processing circuitry 115 via a cache coherent interface, e.g., a CXL interface. The controllers 137 also facilitate data transmission (e.g., RDMA requests) between different servers 105 by bypassing the processing circuitry 115. The ToR Ethernet switch 110 and the network interface circuitry 125 include an RDMA interface to facilitate RDMA requests between CXL memory devices of different servers (e.g., the ToR Ethernet switch 110 and the network interface circuitry 125 provide hardware offload or acceleration of RDMA over Converged Ethernet (RoCE), Infiniband, and iWARP packets).

[0038] The CXL interconnect of the system follows a cache coherent protocol such as the CXL 1.1 standard, or in one embodiment, the CXL 2.0 standard, a future version of CXL, or any other suitable protocol (e.g., a cache coherent protocol). Memory modules 135 are attached directly to processing circuitry 115 as shown, and ToR Ethernet switches 110 are used to scale the system to a larger size (e.g., with more servers 105).

[0039] In one embodiment, each server is equipped with multiple direct-attached CXL memory modules 135, as shown in FIG. 1A. Each memory module 135 exposes a set of base address registers (BARs) to the host's basic input / output system (BIOS) as a memory range. One or more memory modules 135 contain firmware to transparently manage the memory space behind the host OS map. Each of the memory modules 135 may include any one of a number of memory technologies, including, but not limited to, DRAM, NAND flash, high-bandwidth memory (HBM), and low-power DDR SDRAM (LPDDR SDRAM) technologies, or a combination thereof, and may also include a cache controller or separate partition controllers for memory devices of different technologies (e.g., a memory module 135 that combines several memory devices of different technologies). Each memory module 135 includes a different interface width (x4 to x16) and is configured according to any of a variety of associated form factors, such as U.2, M.2, half height, half length (HHHL), full height, half length (FHHL), E1.S, E1.L, E3.S, and E3.H.

[0040] In one embodiment, as described above, the enhanced CXL switch 130 includes an FPGA (or ASIC) controller 137 to provide additional features beyond switching CXL packets. The enhanced CXL switch 130's controller 137 also acts as a management device for the memory module 135, supporting host control plane processing and enabling rich control semantics and statistics. The controller 137 also includes an additional "backdoor" (e.g., 100 Gigabit Ethernet (GbE)) network interface circuit 125. In one embodiment, the controller 137 resides as a CXL Type 2 device in the processing circuit 115, enabling it to issue cache invalidation instructions to the processing circuit 115 upon receiving a remote write request. In one embodiment, DDIO technology is enabled, with remote data first pooled in the processing circuit's last-level cache (LLC) and then written to the memory module 135. As used herein, a "Type 2" CXL device initiates transactions and implements optional coherent cache and host-managed device memory, and applicable transaction types include all CXL.cache and all CXL.mem transactions.

[0041] As previously mentioned, one or more memory modules 135 include persistent memory or "persistent storage" (i.e., storage in which data is not lost when external power is removed). When memory module 135 exists as persistent storage, controller 137 of memory module 135 manages the persistent domain; for example, controller 137 stores data in persistent storage identified by processing circuitry 115 (e.g., as a result of an application calling a corresponding operating system function) by requesting persistent storage. In such an embodiment, a software API flushes caches and data to persistent storage.

[0042] In one embodiment, direct memory transfers are enabled from the network interface circuit 125 to the memory module 135. Such transfers are one-way transfers to remote memory for high-speed communication in a distributed system. In such an embodiment, the memory module 135 exposes hardware details to the network interface circuit 125 in the system to enable faster RDMA transfers. In such a system, two scenarios occur depending on whether data direct I / O (DDIO) in the processing circuit 115 is enabled or disabled. The DDIO enables direct communication between the Ethernet controller or adapter and the cache of the processing circuit 115. When the DDIO in the processing circuit 115 is enabled, the transfer is targeted to the last-level cache of the processing circuit, from which data is then automatically flushed to the memory module. When the DDIO in the processing circuit 115 is disabled, the memory module 135 operates in a device-bias mode to allow accesses to be received directly by the destination memory module 135 (without the DDIO). An RDMA-enabled network interface circuit 125, including a host channel adapter (HCA), buffers, and other processing, is employed to enable such RDMA transfers, bypassing the target memory buffer transfers present in other modes of RDMA transfers. For example, in such an embodiment, the use of bounce buffers (e.g., buffers in the remote server when the resulting destination in memory is within an address range not supported by the RDMA protocol) is avoided. In one embodiment, RDMA uses other physical media options besides Ethernet (e.g., due to the use of switches configured to handle other network protocols). Examples of RDMA-enabled server-to-server connections include, but are not limited to, InfiniBand (using Ethernet User Datagram Protocol (UDP)), RoCE, and iWARP (using TCP / IP).

[0043] 1B shows a system similar to FIG. 1A in which processing circuit 115 is connected to network interface circuit 125 via memory modules 135. Memory modules 135 and network interface circuit 125 reside in expansion socket adapters 140. Each expansion socket adapter 140 plugs into an expansion socket 145, e.g., an M.2 connector, on the motherboard of server 105. Thus, the server may be any suitable (e.g., industry-standard) server that has been modified by installing expansion socket adapter 140 in expansion socket 145. In such an embodiment, (i) each network interface circuit 125 is integrated into a respective one of memory modules 135, or (ii) each network interface circuit 125 has a PCIe interface (network interface circuit 125 is a PCIe endpoint (i.e., a PCIe slave device)). Thus, the processing circuit 115 connected to the network interface circuit 125 (acting as a PCIe master device, or "loop port") communicates with the network interface circuit 125 via a loop port-to-endpoint PCIe connection, and the controller 137 of the memory module 135 communicates with the network interface circuit 125 via a peer-to-peer PCIe connection.

[0044] According to an embodiment of the present invention, a system is provided that includes a first server having an embedded program processing circuit, a first network interface circuit, and a first memory module, the first memory module including a first memory die and a controller, the controller connected to the first memory die via a memory interface, to the embedded program processing circuit via a cache coherent interface, and to the first network interface circuit. In one embodiment, the first memory module further includes a second memory die, the first memory die including volatile memory and the second memory die including persistent memory. In one embodiment, the persistent memory includes NAND flash. In one embodiment, the controller is configured to provide a flash translation layer for the persistent memory. In one embodiment, the cache coherent interface includes a Compute Express Link (CXL) interface. In one embodiment, the first server includes an expansion socket adapter connected to an expansion socket of the first server, the expansion socket adapter including the first memory module and the first network interface circuit. In one embodiment, the controller of the first memory module is connected to the embedded program processing circuit via the expansion socket. In one embodiment, the expansion socket includes an M.2 socket. In one embodiment, the controller of the first memory module is connected to the first network interface circuit by a peer-to-peer PCIe connection. In one embodiment, the system further includes a second server and a network switch connected to the first server and the second server. In one embodiment, the network switch includes a ToR Ethernet switch. In one embodiment, the controller of the first memory module is configured to receive straight RDMA requests via the network switch and the first network interface circuit and to send straight RDMA responses via the network switch and the first network interface circuit.In one embodiment, the controller of the first memory module is configured to receive a straight RDMA request via the network switch and the first network interface circuit and to transmit a straight RDMA response via the network switch and the first network. In one embodiment, the controller of the first memory module receives data from the second server, stores the data in the first memory module, and transmits a command to the embedded program processing circuit to invalidate cache lines. In one embodiment, the controller of the first memory module includes an FPGA or an ASIC. According to one embodiment of the present invention, there is provided a method for performing remote direct memory access in a computing system, the computing system including a first server and a second server, the first server including an embedded program processing circuit, a network interface circuit, and a first memory module having a controller, the method including receiving a straight RDMA request by the controller of the first memory module and transmitting a straight RDMA response by the controller of the first memory module. In one embodiment, the computing system further includes an Ethernet switch connected to the first server and the second server, and receiving the straight RDMA request includes receiving the straight RDMA request via the Ethernet switch. In one embodiment, the method further includes receiving, by a controller of the first memory module, a read command for a first memory address from the embedded program processing circuit; translating, by the controller of the first memory module, the first memory address to a second memory address; and retrieving, by the controller of the first memory module, data from the first memory module at the second memory address.In one embodiment, the method further includes receiving data by a controller of a first memory module, storing the data in the first memory module by the controller of the first memory module, and sending a command to invalidate cache lines to an embedded program processing circuit by the controller of the first memory. According to one embodiment of the present invention, there is provided a system including a first server having an embedded program processing circuit, a first network interface circuit, and a first memory module, the first memory module including a first memory die and controller means, the controller means connected to the first memory die via a memory interface, connected to the embedded program processing circuit via a cache coherent interface, and connected to the first network interface circuit.

[0045] As shown in FIG. 1C , in one embodiment, a server system includes multiple servers 105 connected together by a ToR Ethernet switch 110. Each server includes one or more processing circuits 115 and is connected to (i) system memory 120 (e.g., DDR4 memory), (ii) one or more network interface circuits 125, and (iii) an enhanced CXL switch 130. The enhanced CXL switch 130 is connected to multiple memory modules 135. That is, the system of FIG. 1C includes a first server 105 having an embedded program processing circuit 115, a network interface circuit 125, a cache coherent switch 130, and a first memory module 135. In the system of FIG. 1C , the first memory module 135 is connected to the cache coherent switch 130 (which is an enhanced CXL switch), which is connected to the network interface circuit 125, and the embedded program processing circuit 115 is connected to the cache coherent switch 130.

[0046] Memory modules 135 are grouped by type, form factor, or technology type (e.g., DDR4, DRAM, LPDDR, HBM, NAND flash, or other persistent storage (e.g., NAND flash)). Each memory module has a CXL interface and includes interface circuitry for translating between CXL packets and signals compatible with the memory in memory module 135. In one embodiment, such interface circuitry is instead in enhanced-function CXL switch 130, and each memory module 135 has an interface that is the native interface of the memory in memory module 135. In one embodiment, enhanced-function CXL switch 130 is integrated into memory module 135 (e.g., integrated into an M.2 form factor package with other components of memory module 135 or integrated into a single integrated circuit).

[0047] The ToR Ethernet switch 110 includes interface hardware to facilitate RDMA requests between aggregate memory devices of different servers. The enhanced CXL switch 130 includes one or more circuits (including, for example, FPGAs or ASICs) to (i) route data to different memory types based on workload, (ii) virtualize host addresses into device addresses, and / or (iii) bypass the processing circuitry 115 to facilitate RDMA requests between different servers.

[0048] The memory modules 135 reside in an expansion box (e.g., in the same rack as the enclosure housing the enclosure's motherboard), which includes a predetermined number (e.g., 20 or more, or 100 or more) of memory modules 135, each connected to an appropriate connector. The modules are in an M.2 form factor, and the connectors are M.2 connectors. In one embodiment, the connection between the servers is via a different network other than Ethernet, e.g., a wireless connection such as Wi-Fi or a 5G connection. Each processing circuit is an x86 processor or another processor, e.g., an ARM processor or a GPU. The PCIe link on which the CXL link is instantiated is PCIe 5.0 or another version (e.g., an earlier version or a later (e.g., future) version (e.g., PCIe 6.0)). In one embodiment, a different cache coherent protocol is used in the system instead of or in addition to CXL, and a different cache coherent switch is used instead of or in addition to the enhanced functionality CXL switch 130. Such a cache coherent protocol may be another standard protocol or a cache coherent variant of a standard protocol (similar to how CXL is a variant of PCIe 5.0). Examples of standard protocols include, but are not limited to, Non-Volatile Dual In-line Memory Module (version P) (NVDIMM-P), Cache Coherent Interconnect for Accelerators (CCIX), and Open Coherent Accelerator Processor Interface (OpenCAPI).

[0049] System memory 120 includes, for example, DDR4 memory, DRAM, HBM, or LPDDR memory. Memory modules 135 are partitioned or include cache controllers for handling multiple memory types. Memory modules 135 come in different form factors, examples of which include, but are not limited to, HHHL, FHHL, M.2, U.2, mezzanine card, daughter card, E1.S, E1.L, E3.L, and E3.S.

[0050] In one embodiment, the system implements an aggregate architecture including multiple servers, each aggregated into multiple CXL-attached memory modules 135. Each memory module 135 contains multiple partitions that are separately exposed as memory devices to multiple processing circuits 115. Each input port of the enhanced CXL switch 130 independently accesses multiple output ports of the enhanced CXL switch 130 and its connected memory modules 135. As used herein, an "input port" or "upstream port" of the enhanced CXL switch 130 is a port connected to (or suitable for connection with) a PCIe loop port, and an "output port" or "downstream port" of the enhanced CXL switch 130 is a port connected to (or suitable for connection with) a PCIe endpoint. As in the embodiment of FIG. 1A, each memory module 135 exposes a collection of base address registers (BARs) to the host BIOS as a memory range. One or more of the memory modules 135 contain firmware for transparently managing memory space behind the host OS map.

[0051] In one embodiment, as described above, enhanced CXL switch 130 includes an FPGA (or ASIC) controller 137 to provide additional functionality beyond switching CXL packets. For example, (as described above) enhanced CXL switch 130 virtualizes memory module 135, i.e., acts as a translation layer, translating between processing-circuit-side addresses (or "processor-side" addresses, i.e., addresses contained in memory read and write commands generated by processing circuitry 115) and memory-side addresses (i.e., addresses used by enhanced CXL switch 130 to address storage locations within memory module 135), thereby masking the physical addresses of memory module 135 and presenting a virtual collection of memory. Controller 137 of enhanced CXL switch 130 also acts as a management device for memory module 135, facilitating host control plane processing. Controller 137 transparently moves data without the involvement of processing circuitry 115 and updates the memory map (or "address translation table") accordingly to ensure subsequent accesses function as expected. The controller 137 includes a switch management device that (i) appropriately binds and unbinds upstream and downstream connections during runtime and (ii) enables rich control semantics and statistics regarding data transfers into and out of the memory module 135. The controller 137 includes an additional “backdoor” 100 GbE or other network interface circuit 125 (in addition to the network interface used to connect to the host) for connecting to other servers 105 or other network equipment. In one embodiment, the controller 137 presents itself to the processing circuit 115 as a Type 2 device, which enables the processing circuit 115 to issue a cache invalidation command when a remote write request is received. In one embodiment, DDIO technology is enabled, and remote data is first pooled in the last level cache (LLC) of the processing circuit 115 and then written (from the cache) to the memory module 135.

[0052] As previously mentioned, one or more memory modules 135 include persistent storage. When memory module 135 exists as a persistent device, controller 137 of enhanced CXL switch 130 manages the persistent domain through processing circuitry 115 by requesting persistent storage (e.g., storing identified data in persistent storage through the use of corresponding operating system functions). In such an embodiment, a software API flushes caches and data to persistent storage.

[0053] In one embodiment, direct memory transfer to memory module 135 has operations performed by the controller of memory module 135 performed by controller 137 of enhanced functionality CXL switch 130, and may be performed in a manner similar to that described with respect to the embodiment of Figures 1A and 1B.

[0054] As mentioned above, in one embodiment, memory modules 135 are organized into groups, e.g., one group that is memory intensive, another group that is HBM heavy, another group that has limited density and performance, and a different group that is high density. Such groups may have different form factors or be based on different technologies. Controller 137 of enhanced CXL switch 130 intelligently routes data and commands based on, for example, workload, tagging, or quality of service (QoS). For read requests, there may be no routing based on such factors.

[0055] The controller 137 of the enhanced CXL switch 130 also virtualizes processing circuit-side addresses and memory-side addresses, as described above, allowing the controller 137 of the enhanced CXL switch 130 to determine where data is stored. The controller 137 of the enhanced CXL switch 130 makes such decisions based on information or instructions received from the processing circuit 115. For example, an operating system may provide a memory allocation function that allows an application to specify that low-latency storage, high-bandwidth storage, or persistent storage be allocated, and such application-initiated requests are taken into account by the controller 137 of the enhanced CXL switch 130 when determining where (on one of the memory modules 135) to allocate memory. For example, storage for which high bandwidth is required by the application may be allocated to memory modules 135 including HBM, storage for which data persistence is required by the application may be allocated to memory modules 135 including NAND flash, and other storage (not required by the application) may be stored in memory modules 135 including relatively inexpensive DRAM. In one embodiment, controller 137 of enhanced CXL switch 130 makes decisions about where to store particular data based on network usage patterns. For example, controller 137 of enhanced CXL switch 130 determines by monitoring usage patterns that data within a particular range of physical addresses is accessed more frequently than other data, and controller 137 of enhanced CXL switch 130 copies this data to memory module 135 that includes HBM and modifies the address translation table so that the data in the new location is stored in the same range of virtual addresses. In one embodiment, one or more memory modules 135 include flash memory (e.g., NAND flash), and controller 137 of enhanced CXL switch 130 implements a flash translation layer for this flash memory.The flash translation layer supports overwriting processor-side memory locations (by moving data to a different location and invalidating the data's previous location) and performs garbage collection (e.g., deleting any valid data in a block after moving it to another block if the portion of data in the block marked as invalid exceeds a threshold).

[0056] In one embodiment, the controller 137 of the enhanced CXL switch 130 facilitates physical function (PF) to PF transfers. For example, if one of the processing circuits 115 needs to move data from one physical address to another (which have the same virtual address, and this fact need not affect the operation of the processing circuit 115), or if the processing circuit 115 needs to move data between two virtual addresses (which the processing circuit 115 needs to have), the controller 137 of the enhanced CXL switch 130 can oversee the transfer without the processing circuit 115's involvement. For example, the processing circuit 115 sends a CXL request, and the data does not go to the processing circuit 115 but is sent from one memory module 135 to another memory module 135 after the enhanced CXL switch 130 (e.g., the data is copied from one memory module 135 to another memory module 135). In this situation, because the processing circuit 115 initiated the CXL request, the processing circuit 115 needs to flush its cache to ensure consistency. Alternatively, if a Type 2 memory device (e.g., one of the memory modules 135 or an accelerator that is also connected to the CXL switch) initiates a CXL request and the switch is not virtualized, the Type 2 memory device sends a message to the processing circuit 115 to invalidate its cache.

[0057] In one embodiment, controller 137 of enhanced CXL switch 130 facilitates RDMA requests between servers. Remote server 105 initiates such RDMA requests, which are transmitted through ToR Ethernet switch 110 to reach enhanced CXL switch 130 of server 105 (the “local server”), which responds to the RDMA request. Enhanced CXL switch 130 is configured to receive such RDMA requests and treats a group of memory modules 135 in receiving server 105 (i.e., the server receiving the RDMA request) as its own memory space. At the local server, enhanced CXL switch 130 receives the RDMA request as a direct RDMA request (i.e., an RDMA request that is not routed through the local server's processing circuitry 115) and sends a direct response to the RDMA request (i.e., sends the response without being routed through the local server's processing circuitry 115). At the remote server, the response (e.g., data sent by the local server) is received by the remote server's enhanced function CXL switch 130 and stored in the remote server's memory module 135 without being routed through the remote server's processing circuitry 115.

[0058] 1C , in which processing circuitry 115 is connected to network interface circuitry 125 via enhanced functionality CXL switch 130. Enhanced functionality CXL switch 130, memory module 135, and network interface circuitry 125 reside in expansion socket adapter 140. Expansion socket adapter 140 is a circuit board or module that plugs into an expansion socket, e.g., PCIe connector 145, on the motherboard of server 105. Thus, the server may be any suitable server that is modified only by installing expansion socket adapter 140 in PCIe connector 145. Memory module 135 is installed in a connector (e.g., an M.2 connector) of expansion socket adapter 140. In such an embodiment, (i) network interface circuits 125 are integrated into enhanced functionality CXL switch 130, or (ii) each network interface circuit 125 has a PCIe interface (network interface circuits 125 are PCIe endpoints), so that processing circuits 115 connected to network interface circuits 125 communicate with network interface circuits 125 via root port-to-endpoint PCIe connections. Controller 137 of enhanced functionality CXL switch 130 (which has PCIe input ports connected to processing circuits 115 and network interface circuits 125) communicates with network interface circuits 125 via peer-to-peer PCIe connections.

[0059] According to an embodiment of the present invention, a system includes a first server having an embedded program processing circuit, a network interface circuit, a cache coherent switch, and a first memory module, the first memory module being connected to the cache coherent switch, the cache coherent switch being connected to the network interface circuit, and the embedded program processing circuit being connected to the cache coherent switch. In one embodiment, the system further includes a second memory module connected to the cache coherent switch, the first memory module including volatile memory and the second memory module including persistent memory. In one embodiment, the cache coherent switch is configured to virtualize the first memory module and the second memory module. In one embodiment, the first memory module includes flash memory, and the cache coherent switch is configured to provide a flash translation layer for the flash memory. In one embodiment, the cache coherent switch monitors an access frequency of a first memory location in the first memory module, determines whether the access frequency exceeds a first threshold, and copies the contents of the first memory location to a second memory location in the second memory module. In one embodiment, the second memory module includes HBM. In one embodiment, the cache coherent switch maintains a table for mapping processor-side addresses to memory-side addresses. In one embodiment, the system further includes a second server and a network switch connected to the first server and the second server. In one embodiment, the network switch includes a ToR Ethernet switch. In one embodiment, the cache coherent switch is configured to receive straight RDMA requests and transmit straight RDMA responses. In one embodiment, the cache coherent switch is configured to receive remote RDMA requests via the ToR Ethernet switch and the network interface circuitry and transmit straight RDMA responses via the ToR Ethernet switch and the network interface circuitry.In one embodiment, the cache coherent switch is configured to support the CXL protocol. In one embodiment, the first server includes an expansion socket adapter connected to an expansion socket of the first server, the expansion socket adapter including a cache coherent switch and a memory module socket, and the first memory module is connected to the cache coherent switch via the memory module socket. In one embodiment, the memory module socket includes an M.2 socket. In one embodiment, the network interface circuitry is on the expansion socket adapter. According to one embodiment of the present invention, there is provided a method for performing remote direct memory access in a computing system, the computing system including a first server and a second server, the first server having an embedded program processing circuit, a network interface circuit, a cache coherent switch, and a first memory module, the method including receiving a straight RDMA request by the cache coherent switch and transmitting a straight RDMA response by the cache coherent switch. In one embodiment, the computing system further includes an Ethernet switch, and receiving the straight RDMA request includes receiving the straight RDMA request via the Ethernet switch. In one embodiment, the method further includes receiving, by the cache coherent switch, a read command for a first memory address from the embedded program processing circuit, translating, by the cache coherent switch, the first memory address to a second memory address, and retrieving, by the cache coherent switch, data from the first memory module at the second memory address. In one embodiment, the method further includes receiving, by the cache coherent switch, the data, by the cache coherent switch, storing, by the cache coherent switch, the data in the first memory module, and sending, by the cache coherent switch, a command to invalidate a cache line to the embedded program processing circuit.According to an embodiment of the present invention, a system is provided that includes a first server having an embedded program processing circuit, a network interface circuit, a cache coherent switching means, and a first memory module, wherein the first memory module is connected to the cache coherent switching means, the cache coherent switching means is connected to the network interface circuit, and the embedded program processing circuit is connected to the cache coherent switching means.

[0060] FIG. 1E illustrates an embodiment in which multiple servers 105 are each connected to a server link switch 112, which is a ToR PCIe5 CXL switch with PCIe capabilities, as shown. The server link switch 112 is referred to as a ToR server link switch. The server link switch 112 includes an FPGA or ASIC and provides superior performance (in terms of throughput and latency) over Ethernet switches. Each server 105 includes an enhanced functionality CXL switch 130 and multiple memory modules 135 connected to the server link switch 112 via multiple PCIe connectors. Each of the servers 105 also includes one or more processing circuits 115 and a system memory 120, as shown. The server link switch 112 acts as a master, and each of the enhanced functionality CXL switches 130 acts as a slave, as described in more detail below.

[0061] In the embodiment of Figure 1E, the server link switch 112 groups or batches multiple cache requests received from different servers 105, grouping packets to reduce control overhead. The enhanced CXL switch 130 (i) routes data to different memory types based on workload, (ii) virtualizes processor-side addresses into memory-side addresses, and (iii) includes a slave controller (e.g., a slave FPGA or a slave ASIC) to bypass the processing circuitry 115 and facilitate coherent requests between different servers 105. The system shown in Figure 1E is CXL 2.0-based, includes distributed shared memory within the rack, and uses the server link switch 112 to naturally connect to remote nodes.

[0062] The server link switch 112 has additional network connections (e.g., an Ethernet connection as shown, or another type of connection, e.g., a wireless connection such as a Wi-Fi connection or a 5G connection) for connecting to other servers or clients. The server link switch 112 and the enhanced functionality CXL switch 130 each include a controller that is or has processing circuitry, such as an ARM processor. The PCIe interface may comply with the PCIe 5.0 standard, or an earlier or future version of the PCIe standard, or an interface that complies with another standard (e.g., NVDIMM-P, CCIX, or OpenCAPI) may be used instead of a PCIe interface. The memory module 135 may include various memory types, including DDR4 DRAM, HBM, LPDDR, NAND flash, or solid state drives (SSDs). The memory module 135 may be partitioned or may include a cache controller to handle multiple memory types and may be in different form factors such as HHHL, FHHL, M.2, U.2, mezzanine card, daughter card, E1.S, E1.L, E3.L, or E3.S.

[0063] In the embodiment of FIG. 1E, the enhanced CXL switch 130 enables one-to-many and many-to-one switching, enabling a fine-grained load-store interface at the flit (64-byte) level. Each server has an aggregated memory device, and each device is divided into multiple logical devices with their own LD-ID. The server link switch 112 enables the one-to-many functionality, while the enhanced CXL switch 130 of the server 105 enables the many-to-one functionality. The server link switch 112 may be a PCIe switch, a CXL switch, or both. In such a system, the requesters are the processing circuits 115 of the multiple servers 105, and the responders are the multiple aggregated memory modules 135. A layer of two switches (with a master switch, the server link switch 112, as described above, and a slave switch, the enhanced CXL switch 130) enables all communication. Each memory module 135 contains one physical function (PF) and up to 16 separate logical devices. In one embodiment, the number of logical devices (e.g., partitions) is limited (e.g., to 16), and there is also one control partition (which is the physical function used to control the device). Each memory module 135 is a Type 2 device with implementations of cxl.cache, cxl.mem, and cxl.io, as well as an Address Translation Service (ATS) to handle cache line copies held by the processing circuitry 115. The enhanced CXL switch 130 and fabric manager controls the discovery of memory modules 135, (i) performs device discovery and virtual CXL software creation, and (ii) binds virtual ports to physical ports. As in the embodiment of Figures 1A-1D, the fabric manager operates via a connection over SMBus sideband.An interface to memory module 135 that is an Intelligent Platform Management Interface (IPMI) or Redfish standard compliant interface (and may provide additional functionality not required by the standard) allows for composability.

[0064] As noted above, some embodiments implement a hierarchy with a master controller (implemented in an FPGA or ASIC) that is part of the server link switch 112 and slave controllers that are part of the enhanced CXL switch 130 to provide a rod-store interface (i.e., an interface that has a granularity of a cache line (e.g., 64 bytes) and operates within the coherence domain without the involvement of software drivers). Such a rod-store interface extends the coherence domain beyond an individual server, CPU, or host and includes electrical or optical physical media (e.g., electro-optical transceiver optical connections at both ends). During operation, the master controller (in the server link switch 112) boots (or "reboots") and configures all servers 105 on the rack. The master controller has visibility into all hosts, (i) discovers each server and discovers the number of servers 105 and memory modules 135 present in the server cluster, (ii) configures each server 105 independently, (iii) enables or disables some blocks of memory in different servers based on the rack configuration (e.g., enables or disables one of the memory modules 135), (iv) controls access (e.g., one server controls another), (v) provides flow control (e.g., sends data from one server to another and performs flow control on the data since all host and device requests go through the master), (vi) groups or batches requests or packets (e.g., multiple cache requests received from different servers 105 by the master), and (vii) receives remote software updates, broadcast communications, etc. In batch mode, the server link switch 112 receives multiple packets destined for the same server (e.g., destined for the first server) and sends them together (i.e., without stopping between them) to the first server. For example, the server link switch 112 receives a first packet from a second server, receives a second packet from a third server, and sends both the first packet and the second packet to the first server.Each server 105 exposes to the master controller (i) an IPMI network interface, (ii) a system event log (SEL), and (iii) a board management controller (BMC) to enable the master controller to instantly measure performance, measure reliability, and reconfigure the server 105.

[0065] In one embodiment, a software architecture is used that facilitates a highly available rod-to-store interface. Such a software architecture provides reliability, replication, consistency, system coherence, hashing, caching, and durability. The software architecture provides reliability (in systems with a large number of servers) by performing periodic hardware checks of CXL device components via IPMI. For example, the server link switch 112 queries the status of the memory server 150 via the IPMI interface, such as power status (whether the power supply to the memory server 150 is operating properly), network status (whether the interface to the server link switch 112 is operating properly), and error checking status (whether an error condition exists in any subsystem of the memory server 150). The software architecture provides replication in that the master controller can replicate data stored in the memory modules 135 and maintain data consistency between the replicas.

[0066] The software structure provides consistency in that the master controllers are configured with different consistency levels, and the server link switch 112 adjusts the packet format according to the consistency level it maintains. For example, if eventual consistency is maintained, the server link switch 112 reorders requests, while for strict consistency, the server link switch 112 maintains a scoreboard of all requests with precise timestamps at the switch. The software structure provides system consistency in that multiple processing circuits 115 can read from or write to the same memory address, and the master controller is responsible for reaching the home node of the address (using a directory lookup) or broadcasting the request on a common bus to maintain consistency.

[0067] The software structure provides hashing in that the server link switch 112 and enhanced CXL switch 130 maintain a virtual mapping of addresses using multiple hash functions and consistent hashing to map data evenly to all CXL devices across all nodes at startup (or to adjust when one server goes down or up). The software structure provides caching in that the master controller designates a particular memory partition (e.g., in a memory module 135 including HBM or similar technology) to act as a cache (e.g., utilizing a write-through or write-back cache). The software structure provides persistence in that the master and slave controllers manage persistent domains and flash.

[0068] In one embodiment, the functionality of the CXL switch 130 is integrated into the controller of the memory module 135. In such an embodiment, the server link switch 112 nevertheless acts as a master and has enhanced functionality as described elsewhere herein. The server link switch 112 can also manage other storage devices in the system, for example, having an Ethernet connection (e.g., a 100 GbE connection) to connect to client machines that are not part of the PCIe network formed by the server link switch 112.

[0069] In one embodiment, the server link switch 112 has enhanced functionality and includes an integrated CXL controller. In another embodiment, the server link switch 112 is merely a physical routing device, and each server 105 includes a master CXL controller. In such an embodiment, the masters of different servers negotiate a master-slave structure. (i) The enhanced functionality CXL switch 130 and (ii) the intelligence of the server link switch 112 are implemented in one or more FPGAs, one or more ASICs, one or more ARM processors, or one or more SSDs with computing capabilities. The server link switch 112 performs flow control, for example, by reordering independent requests. In one embodiment, the interface is load-store, so RDMA is optional, although there are intervening RDMA requests that use the PCIe physical medium (instead of 100 GbE). In such an embodiment, a remote host initiates an RDMA request that is sent to the enhanced functionality CXL switch 130 via the server link switch 112. The server link switch 112 and the enhanced CXL switch 130 prioritize RDMA 4KB requests or CXL flit (64 byte) requests.

[0070] 1C and 1D, an enhanced CXL switch 130 is configured to receive such RDMA requests and treats a group of memory modules 135 of a receiving server 105 (i.e., the server receiving the RDMA request) as its own memory space. The enhanced CXL switch 130 also virtualizes, via processing circuitry 115, and initiates RDMA requests to remote enhanced CXL switches 130 to move data back and forth between servers 105 without the involvement of processing circuitry 115.

[0071] FIG. 1F shows a system similar to that of FIG. 1E, where processing circuitry 115 is connected to network interface circuitry 125 via enhanced functionality CXL switch 130. As in the embodiment of FIG. 1D, in FIG. 1F, enhanced functionality CXL switch 130, memory module 135, and network interface circuitry 125 reside on expansion socket adapter 140. Expansion socket adapter 140 is a circuit board or module that plugs into an expansion socket, such as PCIe connector 145, on the motherboard of server 105. Thus, the server can be any suitable server that is modified only by installing expansion socket adapter 140 into PCIe connector 145. Memory module 135 is installed into a connector (e.g., an M.2 connector) of expansion socket adapter 140. In such an embodiment, (i) the network interface circuits 125 are integrated into the enhanced functionality CXL switch 130, or (ii) each network interface circuit 125 has a PCIe interface (the network interface circuit 125 is a PCIe endpoint) and the processing circuit 115 connected thereto communicates with the network interface circuit 125 via a root port to endpoint PCIe connection, and the controller 137 of the enhanced functionality CXL switch 130 (which has a PCIe input port connected to the processing circuit 115 and the network interface circuit 125) communicates with the network interface circuit 125 via a peer-to-peer PCIe connection.

[0072] According to one embodiment of the present invention, a system includes a first server having an embedded program processing circuit, a cache coherent switch, and a first memory module, a second server, and a server link switch connected to the first server and the second server, wherein the first memory module is connected to the cache coherent switch, the cache coherent switch is connected to the server link switch, and the embedded program processing circuit is connected to the cache coherent switch. In one embodiment, the server link switch includes a PCIe switch. In one embodiment, the server link switch includes a CXL switch. In one embodiment, the server link switch includes a ToR CXL switch. In one embodiment, the server link switch is configured to discover the first server. In one embodiment, the server link switch is configured to reboot the first server. In one embodiment, the server link switch is configured to cause the cache coherent switch to disable the first memory module. In one embodiment, the server link switch is configured to transmit data from the second server to the first server and perform flow control on the data. In one embodiment, the system further includes a third server connected to the server link switch, where the server link switch receives a first packet from the second server and a second packet from the third server, and transmits the first packet and the second packet to the first server. In one embodiment, the system further includes a second memory module connected to the cache coherent switch, where the first memory module includes volatile memory and the second memory module includes persistent memory. In one embodiment, the cache coherent switch is configured to virtualize the first memory module and the second memory module. In one embodiment, the first memory module includes flash memory, and the cache coherent switch is configured to provide a flash translation layer for the flash memory.In one embodiment, the first server includes an expansion socket adapter connected to an expansion socket of the first server, the expansion socket adapter including a cache coherent switch and a memory module socket, and the first memory module is connected to the cache coherent switch via the memory module socket. In one embodiment, the memory module socket includes an M.2 socket. In one embodiment, the cache coherent switch is connected to the server link switch via a connector, the connector being on the expansion socket adapter. According to one embodiment of the present invention, there is provided a method for performing remote direct memory access in a computing system, the computing system including a first server, a second server, a third server, and a server link switch connected to the first server, the second server, and the third server, the first server including an embedded program processing circuit, a cache coherent switch, and a first memory module, the method including receiving a first packet from the second server via the server link switch, receiving a second packet from the third server via the server link switch, and transmitting the first packet and the second packet to the first server. In one embodiment, the method further includes receiving, by the cache coherent switch, a straight RDMA request and transmitting, by the cache coherent switch, a straight RDMA response. In one embodiment, receiving the straight RDMA request includes receiving the straight RDMA request through a server link switch. In one embodiment, the method further includes receiving, by the cache coherent switch, a read command for a first memory address from an embedded program processing circuit, translating, by the cache coherent switch, the first memory address to a second memory address, and retrieving, by the cache coherent switch, data from the first memory module at the second memory address.According to one embodiment of the present invention, a system includes a first server including an embedded program processing circuit, a cache coherent switching means, and a first memory module, a second server, and a server link switch connected to the first server and the second server, wherein the first memory module is connected to the cache coherent switching means, the cache coherent switching means is connected to the server link switch, and the embedded program processing circuit is connected to the cache coherent switching means.

[0073] FIG. 1G illustrates an embodiment in which multiple memory servers 150 are each connected to a server link switch 112, which is a PCIe 5.0 CXL switch, as shown. As in the embodiments of FIGS. 1E and 1F, the server link switch 112 includes an FPGA or an ASIC to provide better performance (in terms of throughput and latency) than an Ethernet switch. As in the embodiments of FIGS. 1E and 1F, the memory server 150 includes multiple memory modules 135 connected to the server link switch 112 via multiple PCIe connectors. In the embodiment of FIG. 1G, the processing circuitry 115 and system memory 120 may be absent, and the primary purpose of the memory server 150 is to provide memory for use by other servers 105 with computing resources.

[0074] As shown in FIG. 1G, the server link switch 112 groups or batches multiple cache requests received from different memory servers 150, grouping packets to reduce control overhead. The enhanced CXL switch 130 includes configurable hardware building blocks for (i) routing data to different memory types based on workload and (ii) virtualizing processor-side addresses (translating processor-side addresses into memory-side addresses). The system shown in FIG. 1G is CXL 2.0-based, includes configurable disaggregated shared memory within the rack, and uses the server link switch 112 to provide pooled (i.e., aggregated) memory to remote devices.

[0075] The server link switch 112 has additional network connections (e.g., an Ethernet connection as shown, or another type of connection, e.g., a wireless connection such as a Wi-Fi connection or 5G connection) for connecting to other servers or clients. The server link switch 112 and the enhanced CXL switch 130 each include a controller that is or has a processing circuit, such as an ARM processor. The PCIe interface may comply with the PCIe 5.0 standard, an earlier or future version of the PCIe standard, or may use another standard (e.g., NVDIMM-P, CCIX, or OpenCAPI) instead of PCIe. The memory module 135 includes various memory types, including DDR4 DRAM, HBM, LPDDR, NAND flash, and solid-state drives (SSDs). The memory module 135 may be partitioned or include a cache controller to handle multiple memory types and may be in different form factors, such as HHHL, FHHL, M.2, U.2, mezzanine card, daughter card, E1.S, E1.L, E3.L, or E3.S.

[0076] In the embodiment of FIG. 1G, the enhanced CXL switch 130 enables one-to-many and many-to-one switching, enabling a fine-grain rod-store interface at the flit (64-byte) level. Each memory server 150 has a collection of memory devices, each partitioned into multiple logical devices with their own LD-ID. The enhanced CXL switch 130 includes a controller 137 (e.g., an ASIC or FPGA) and circuitry (either separate from or part of the ASIC or FPGA) for device discovery, enumeration, partitioning, and presenting physical address ranges. Each memory module 135 has one physical function (PF) and up to 16 separate logical devices. In one embodiment, the number of logical devices (e.g., the number of partitions) is limited (e.g., to 16), and there is also one control partition (which is the physical function used to control the devices). Each memory module 135 is a Type 2 device with implementations of cxl.cache, cxl.mem, and cxl.io, as well as an Address Translation Service (ATS), to handle cache line copies held by the processing circuitry 115.

[0077] The enhanced CXL switch 130 and fabric manager controls the discovery of memory modules 135, (i) performs device discovery and creation of virtual CXL software, and (ii) binds virtual ports to physical ports. As in the embodiment of Figures 1A-1D, the fabric manager operates via a connection over SMBus sideband. The interface to memory modules 135, which is an interface that conforms to the IPMI or Redfish standards (and may provide additional functionality not required by the standards), allows for configurability.

[0078] The building blocks for the embodiment of FIG. 1G are implemented in an FPGA or ASIC (as described above) and include a CXL controller 137 that switches memory devices (e.g., memory module 135), SSDs, accelerators (GPUs, NICs), CXL and PCIe5 connectors, and a collection of firmware to enable exposing device details to the operating system's Advanced Configuration and Power Interface (ACPI) tables, such as the Heterogeneous Memory Attribute Table (HMAT) or Static Resource Affinity Table (SRAT).

[0079] In one embodiment, the system provides configurability. Based on software configuration, the system provides the ability to online and offline CXL devices and other accelerators, and groups accelerator, memory, and storage device resources and assigns them to each memory server 150 in the rack. The system hides the physical address space and provides a transparent cache using faster devices such as HBM and SRAM.

[0080] In the embodiment of FIG. 1G, the controller 137 of the enhanced CXL switch 130 (i) manages the memory modules 135, (ii) aggregates and controls heterogeneous devices such as NICs, SSDs, GPUs, and DRAMs, and (iii) performs dynamic storage reconfiguration to the memory devices through power gating. For example, the server link switch 112 disables (i.e., powers down or reduces) power to one of the memory modules 135 (by instructing the enhanced CXL switch 130 to disable power to the memory module 135). The enhanced CXL switch 130 disables power to the memory module 135 when instructed by the server link switch 112 to disable power to the memory module 135. Such disabling can conserve power and improve performance (e.g., throughput and latency) of other memory modules 135 in the memory server 150. Each remote server 105 sees a different logical view of the memory modules 135 and their connections based on negotiations. The controller 137 of the enhanced CXL switch 130 maintains state to ensure each remote server maintains its allocated resources and connections, and performs memory compression or deduplication (using configurable chunk sizes) to conserve memory capacity. The disaggregated rack of FIG. 1G has its own BMC. It also exposes an IPMI network interface and system event log (SEL) to remote devices, allowing a master (e.g., a remote server using storage provided by memory server 150) to measure performance and reliability on the fly and reconfigure the disaggregated rack. In a manner similar to that described herein for the embodiment of FIG. 1E, the disaggregated rack of FIG. 1G provides reliability, replication, consistency, system coherence, hashing, caching, and durability. For example, coherence is provided to multiple remote servers reading from or writing to the same memory address, with each remote server configured with a different consistency level.In one embodiment, the server link switch 112 maintains eventual consistency between data stored in a first memory server and data stored in a second memory server. The server link switch 112 maintains different consistency levels for different server pairs. For example, the server link switch 112 also maintains strict consistency, sequential consistency, causal consistency, or processor consistency between data stored in a first memory server and data stored in a third memory server. The system uses communication in the "local band" (server link switch 112) and "global band" (disaggregated server) domains. Writes are flushed to the "global band" to acknowledge new reads from other servers. The controller 137 of the enhanced CXL switch 130 manages the persistence domains and flushes them separately to each remote server. For example, a cache-coherent switch can monitor the fullness of a first region of memory (volatile memory operating as a cache). When the fullness level exceeds a threshold, the cache-coherent switch moves data from the first region of memory to the second region of memory. The second region of memory is in persistent memory. Flow control can be handled in that priorities are set by the controller 137 of the enhanced CXL switch 130 between remote servers, providing different perceived latencies and bandwidths.

[0081] According to an embodiment of the present invention, a system includes a first memory server having a cache coherent switch and a first memory module, a second memory server, and a server link switch connected to the first memory server and the second memory server, wherein the first memory module is connected to the cache coherent switch, and the cache coherent switch is connected to the server link switch. In one embodiment, the server link switch is configured to disable power to the first memory module. In one embodiment, the server link switch is configured to disable power to the first memory module by instructing the cache coherent switch to disable power to the first memory module, and the cache coherent switch is configured to disable power to the first memory module when instructed by the server link switch to disable power to the first memory module. In one embodiment, the cache coherent switch is configured to perform deduplication in the first memory module. In one embodiment, the cache coherent switch is configured to compress data and store the compressed data in the first memory module. In one embodiment, the server link switch is configured to query a status of the first memory server. In one embodiment, the server link switch is configured to query the state of the first memory server via IPMI. In one embodiment, querying the state includes querying a state selected from the group consisting of a power state, a network state, and an error checking state. In one embodiment, the server link switch is configured to batch cache requests directed to the first memory server. In one embodiment, the system further includes a third memory server connected to the server link switch, the server link switch configured to maintain consistency between data stored in the first memory server and data stored in the third memory server, the consistency level being selected from the group consisting of strict consistency, sequential consistency, causal consistency, and processor consistency.In one embodiment, the cache coherent switch is configured to monitor the fullness of a first region of memory and move data from the first region of memory to a second region of memory, where the first region of memory is in volatile memory and the second region of memory is in persistent memory. In one embodiment, the server link switch includes a PCIe switch. In one embodiment, the server link switch includes a CXL switch. In one embodiment, the server link switch includes a top-of-rack (ToR) CXL switch. In one embodiment, the server link switch is configured to transmit data from the second memory server to the first memory server and perform flow control on the data. In one embodiment, the system further includes a third memory server connected to the server link switch, where the server link switch is configured to receive a first packet from the second memory server, receive a second packet from the third memory server, and transmit the first packet and the second packet to the first memory server. According to one embodiment of the present invention, there is provided a method for performing RDMA in a computing system, the computing system including a first memory server, a first server, a second server, and a server link switch connected to the first memory server, the first server, and the second server, the first memory server including a cache coherent switch and a first memory module, the first server including an embedded program processing circuit, and the second server including an embedded program processing circuit, the method including: receiving a first packet from the first server by the server link switch; receiving a second packet from the second server by the server link switch; and transmitting the first packet and the second packet to the first memory server. In one embodiment, the method further includes compressing data by the cache coherent switch and storing the data in the first memory module. In one embodiment, the method further includes querying a status of the first memory server by the server link switch.According to one embodiment of the present invention, a system includes a first memory server including a cache coherent switch and a first memory module, a second memory server, and a server link switching means connected to the first memory server and the second memory server, wherein the first memory module is connected to the cache coherent switch and the cache coherent switch is connected to the server link switching means.

[0082] 2A-2D are flowcharts of various embodiments. In embodiments of these flowcharts, the processing circuitry 115 is a CPU, while in other embodiments, it is another processing circuit (e.g., a GPU). Referring to FIG. 2A, the controller 137 of the memory module 135 of any of the embodiments of FIGS. 1A and 1B, or the enhanced CXL switch 130 of any of the embodiments of FIGS. 1C-1G, virtualizes through the processing circuitry 115 and initiates an RDMA request with the enhanced CXL switch 130 of another server 105 to move data back and forth between the servers 105 (with the virtualization handled by the controller 137 of the enhanced CXL switch 130) without involving the processing circuitry 115 in either server. For example, at S205, the controller 137 of the memory module 135 or the enhanced CXL switch 130 generates an RDMA request for additional remote memory (e.g., CXL memory or collective memory). At S210, the controller 137 of the memory module 135 or the enhanced CXL switch 130 sends the request to the ToR Ethernet switch 110 (which has an RDMA interface), bypassing the processing circuitry 115. At S215, the ToR Ethernet switch 110 routes the RDMA request to the remote server 105 via RDMA access to the remote collective memory, bypassing the remote processing circuitry 115. At S220, the ToR Ethernet switch 110 receives the processed data and routes the data via RDMA to the local memory module 135 or the local enhanced CXL switch 130, bypassing the local processing circuitry 115. At S222, the controller 137 of the memory module 135 or the enhanced CXL switch 130 in the embodiment of Figures 1A and 1B receives the RDMA response directly (e.g., without being forwarded by the processing circuitry 115).

[0083] In such an embodiment, the controller 137 of the remote memory module 135 or the enhanced CXL switch 130 of the remote server 105 is configured to receive a straight RDMA request and send a straight RDMA response. As used herein, receiving a “straight RDMA request” (or receiving such a request “straight”) by the controller 137 of the remote memory module 135 or the enhanced CXL switch 130 means receiving such a request by the controller 137 of the remote memory module 135 or the enhanced CXL switch 130 without being forwarded or processed by the processing circuitry 115 of the remote server 105, and sending a “straight RDMA response” (or sending such a request “straight”) by the controller 137 of the remote memory module 135 or the enhanced CXL switch 130 means sending such a response without being forwarded or processed by the processing circuitry 115 of the remote server 105.

[0084] 2B , in another embodiment, RDMA is performed with processing circuitry 115 of a remote server 105 involved in data processing. For example, at S225, processing circuitry 115 transmits a data or workload request over Ethernet. At S230, ToR Ethernet switch 110 receives the request and routes it to a corresponding server 105 among the plurality of servers 105. At S235, the request is received within the server via a port of network interface circuitry 125 (e.g., a 100 GbE-capable NIC). At S240, processing circuitry 115 (e.g., an x86 processing circuit) receives the request from network interface circuitry 125 (e.g., a PCIe connector). At S245, processing circuitry 115 processes the request using (e.g., both) DDR and additional memory resources via the CXL 2.0 protocol to share memory (which in the embodiment of FIGS. 1A and 1B is collective memory).

[0085] 2C , in the embodiments of FIGS. 1E and 1F , RDMA is performed with the processing circuitry 115 of the remote server 105 participating in data processing. For example, at S225, the processing circuitry 115 sends a data or workload request via Ethernet or PCIe. At S230, the ToR PCIe5 CXL switch 3.112 (server link switch) receives the request and routes it to a corresponding server 105 among the multiple servers 105. At S235, the request is received within the server via a port of the PCIe connector 125. At S240, the processing circuitry 115 (e.g., an x86 processing circuit) receives the request from the network interface circuitry 125 (e.g., a PCIe connector). At S245, the processing circuitry 115 processes the request using (e.g., together) DDR and additional memory resources via the CXL 2.0 protocol to share memory (which in the embodiments of FIGS. 1A and 1B is collective memory). At S250, the processing circuit 115 identifies a requirement to access memory content (e.g., content of a DDR or collective memory) from a different server. At S252, the processing circuit 115 sends a request for memory content (e.g., content of a DDR or collective memory) from the different server via a CXL protocol (e.g., CXL 1.1 or CXL 2.0). At S254, the request is propagated to the ToR PCIe5 switch 112 via a local PCIe connector, which then sends the request to a second PCIe connector of a second server on the rack. At S256, a second processing circuit 115 (e.g., an x86 processing circuit) receives the request from the second PCIe connector. At S258, the second processing circuit 115 processes the request (e.g., retrieval of memory content) together using a second DDR and a second additional memory resource via the CXL 2.0 protocol to share the collective memory. At S260, the second processing circuit (eg, x86 processing circuit) retransmits the results of the request to the original processing circuit via the respective PCIe connectors and the ToR PCIe5 switch 112.

[0086] 2D , in the embodiment of FIG. 1G , RDMA is performed in the processing circuitry 115 of a remote server 105 involved in data processing. For example, at S225, the processing circuitry 115 sends a data or workload request over Ethernet. At S230, the ToR PCIe5 CXL switch 112 (server link switch) receives the request and routes it to a corresponding server 105 among the multiple servers 105. At S235, the request is received within the server via a port of the network interface circuitry 125 (e.g., a 100 GbE-compatible NIC). At S262, the memory module 135 receives the request from the PCIe connector. At S264, the controller 137 of the memory module 135 processes the request using local memory. At S250, the controller 137 of the memory module 135 identifies requirements for accessing memory contents (e.g., contents of a collective memory) from different servers. At S252, the controller 137 of the memory module 135 sends a request for memory content (e.g., the contents of the collective memory) from a different server via the CXL protocol. At S254, the request is propagated via a local PCIe connector to the ToR PCIe5 CXL switch 112, which then sends the request to a second PCIe connector of a second server on the rack. At S266, the second PCIe connector provides access via the CXL protocol to share the collective memory so that the controller 137 of the memory module 135 can retrieve the memory content.

[0087] As used herein, a "server" is a computing system that includes at least one stored-program processing circuit (e.g., processing circuit 115), at least one memory resource (e.g., system memory 120), and at least one circuit for providing network connectivity (e.g., network interface circuit 125). As used herein, "portion" means "at least a portion" of something, and thus means all or less than all of something. Thus, a "portion" of something includes the entire thing as a special case, i.e., an instance in which the entire thing is part of the thing.

[0088] The background provided in the Background section of this specification is included for context purposes only, and the contents of this section are not admitted to be prior art. Any component or combination of components illustrated (e.g., in any system diagrams included herein) may be used to perform one or more of the operations of any flowcharts included herein. Also, (i) the operations are example operations and may include various additional steps not explicitly described, and (ii) the temporal order of the operations may be changed.

[0089] The terms "processing circuit" or "controller means" are used herein to refer to any combination of hardware, firmware, and software used to process data or digital signals. Processing circuit hardware includes, for example, programmable logic devices such as ASICs, general-purpose or special-purpose CPUs, digital signal processors (DSPs), GPUs, and FPGAs. In a processing circuit, as used herein, each function is performed by hardware configured to perform that function, i.e., hardwired, or by more general-purpose hardware, such as a CPU, configured to execute instructions stored on a non-transitory storage medium. A processing circuit may be fabricated on a single printed circuit board (PCB) or distributed across multiple interconnected PCBs. A processing circuit may include other processing circuits. For example, a processing circuit may include two processing circuits, an FPGA, and a CPU interconnected on a PCB.

[0090] As used herein, a "controller" includes circuitry and may be referred to as a "control circuit" or a "controller circuit." Similarly, a "memory module" may be referred to as a "memory module circuit" or a "memory circuit." As used herein, the term "array" refers to an ordered series of numbers, regardless of how they are stored (e.g., in contiguous memory locations or in a linked list). Where a second number is "within Y%" of a first number, the second number is at least (1-Y / 100) times the first number, and the second number is at most (1+Y / 100) times the first number. As used herein, the term "or" should be interpreted as "and / or," e.g., "A or B" means either "A" or "B," or "A and B."

[0091] As used herein, when a method (e.g., an adjustment) or a first quantity (e.g., a first variable) is described as being "based on" a second quantity (e.g., a second variable), the second quantity is an input to the method or influences the first quantity. For example, the second quantity may be an input (e.g., the only input or one of multiple inputs) to a function that calculates the first quantity, or the first quantity may be the same as the second quantity, or the first quantity may be the same as the second quantity (e.g., stored in the same location in memory).

[0092] In this specification, terms such as "first," "second," and "third" are used to describe various elements, components, regions, layers, and / or sections, but should not be construed as limiting. These terms are used only to distinguish one element, component, region, layer, or section from another element, component, region, layer, or section. Thus, a first element, component, region, layer, or section described in this specification can be referred to as a second element, component, region, layer, or section without departing from the spirit or scope of the inventive concept.

[0093] Various spatially relative terms referring to "below" are used to facilitate the relationship of features to one element or another. Such spatially relative terms are intended to encompass other orientations of the device during use or operation in addition to the orientation depicted in the figures. For example, if the device in the figures is turned over, an element described as "below" another element or feature would then be oriented "above" that other element or feature. Thus, the exemplary term "below" can encompass both above and below. The device may be otherwise oriented (e.g., rotated 90 degrees or at another orientation), and the spatially relative descriptors used herein should be interpreted accordingly. It will also be understood that when a layer is described as being between two layers, it may be the only layer between the two layers, or there may be one or more intermediate layers.

[0094] The terms used herein are intended to describe particular embodiments only and are not intended to limit the scope of the present invention. As used herein, the terms "substantially," "about," and similar terms are used as terms of approximation, not as terms of degree, and are intended to account for inherent variations in measurements or calculations that one of ordinary skill in the art will recognize. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms unless the context clearly dictates otherwise. As used herein, the term "comprising" specifies the presence of disclosed features, numerals, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, numerals, steps, operations, elements, components, and / or groups thereof. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items. Expressions such as "at least one," when preceding a list of elements, modify the entire list of elements, not the individual elements of the list. Furthermore, when used to describe embodiments of the present invention, the term "may" means "one or more embodiments of the present invention." Furthermore, the term "exemplary" is intended to refer to an example or illustration. As used herein, the term "use" is considered synonymous with the term "utilize."

[0095] When an element or layer is described as "on," "connected," "coupled," or "adjacent" another element or layer, it may be directly connected, or there may be one or more other intervening elements or layers. In contrast, when an element or layer is described as "directly on," "directly connected," "directly coupled," or "adjacent" another element or layer, there are no intervening elements or layers.

[0096] Any numerical range recited herein is intended to include all subranges of the same numerical precision subsumed within the recited range. For example, a range "from 1.0 to 10.0" or "between 1.0 and 10.0" is intended to include all subranges between the recited minimum of 1.0 and the recited maximum of 10.0, i.e., including minimums equal to or greater than 1.0 and maximums equal to or less than 10.0, e.g., 2.4 to 7.6. Any maximum numerical limitation recited herein is intended to include all subsumed lower numerical limitations, and any minimum numerical limitation recited herein is intended to include all subsumed higher numerical limitations.

[0097] While exemplary embodiments of systems and methods for managing memory resources have been specifically described and illustrated herein, many modifications and variations will be apparent to those skilled in the art. It should therefore be understood that systems and methods for managing memory resources constructed according to the principles of the present disclosure may be embodied other than as specifically described herein. The invention is also defined in the following claims and their equivalents. [Explanation of symbols]

[0098] 105 Server 110 ToR Ethernet Switch 112 Server Link Switch 115 Processing Circuit 120 system memory 125 Network Interface Circuit 130 Enhanced CXL Switch 135 memory modules 137 Controller 140 Expansion Socket Adapter 145 Expansion Socket (PCIe Connector) 150 Memory Server

Claims

1. a first memory server including a cache coherent switch and a first memory module; a second memory server; a server link switch connected to the first memory server and the second memory server and including a controller; the first memory module is connected to the server link switch via the cache coherent switch; The cache coherent switch includes a controller, and executes processing for the first memory module in response to an instruction from the server link switch.

2. 2. The system of claim 1, wherein the server link switch disables power to the first memory module through the cache coherent switch.

3. the server link switch disables power to the first memory module by instructing the cache coherent switch to disable power to the first memory module; 3. The system of claim 2, wherein the cache coherent switch disables power to the first memory module when instructed by the server link switch to disable power to the first memory module.

4. 10. The system of claim 1, wherein the cache coherent switch performs memory deduplication within the first memory module.

5. 2. The system of claim 1, wherein the cache coherent switch compresses data and stores the compressed data in the first memory module.

6. 2. The system of claim 1, wherein the server link switch queries the state of the first memory module through the cache coherent switch.

7. 2. The system of claim 1, wherein the server link switch includes an Intelligent Platform Management Interface (IPMI) and queries the status of the first memory module via the IPMI.

8. Querying the status includes:

8. The system of claim 7, further comprising querying a state selected from the group consisting of a power state, a network state, and an error checking state.

9. 2. The system of claim 1, wherein the server link switch batches cache requests directed to the first memory server.

10. The system comprises: further comprising a third memory server connected to the server link switch; 2. The system of claim 1, wherein the server link switch maintains a consistency level selected from the group consisting of strict consistency, sequential consistency, causal consistency, and processor consistency between data stored in the first memory server and data stored in the third memory server.

11. The cache coherent switch monitoring the fullness of a first region of memory; moving data from the first region of the memory to a second region of the memory; the first region of memory is in volatile memory; 2. The system of claim 1, wherein the second region of the memory is in non-volatile memory.

12. 10. The system of claim 1, wherein the server link switch comprises a Peripheral Component Interconnect express (PCIe) switch.

13. 10. The system of claim 1, wherein the server link switch comprises a compute express link (CXL) switch.

14. 14. The system of claim 13, wherein the server link switch comprises a top of rack (ToR) CXL switch.

15. 2. The system of claim 1, wherein the server link switch transmits data from the second memory server to the first memory server and performs flow control on the data.

16. The system comprises: further comprising a third memory server connected to the server link switch; The server link switch receiving a first packet from the second memory server; receiving a second packet from the third memory server; 2. The system of claim 1, further comprising transmitting the first packet and the second packet to the first memory server.

17. A method for performing remote direct memory access (RDMA) in a computing system, comprising: The computing system includes: a first memory server including a network interface circuit, a cache coherent switch, and a first memory module; a first server including a stored program processing circuit; a second server including stored program processing circuitry; a server link switch connected to the first memory server, the first server, and the second server, the server link switch including a controller; the server link switch and the network interface circuit include an RDMA interface; the first memory module is connected to the server link switch via the cache coherent switch; the cache coherent switch includes a controller, and executes processing for the first memory module in response to an instruction from the server link switch; The method comprises: receiving a first packet from the first server by the server link switch; receiving a second packet from the second server by the server link switch; transmitting the first packet and the second packet to the first memory server.

18. The method comprises: compressing data by the cache coherent switch; 20. The method of claim 17, further comprising: storing the data in the first memory module.

19. The method comprises:

18. The method of claim 17, further comprising querying the state of the first memory server by the server link switch.

Images