A CXL-based distributed memory crossbar architecture
By connecting host nodes and the memory exchange network using an orthogonal architecture based on CXL, the problem of PCIe bus's inability to transmit data over long distances is solved, enabling efficient memory pool resource sharing and expansion, reducing device power consumption and maintenance costs, and improving system reliability and scalability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA TELECOM CLOUD TECH CO LTD
- Filing Date
- 2023-12-13
- Publication Date
- 2026-05-19
AI Technical Summary
In traditional distributed storage solutions, the PCIe bus cannot achieve long-distance transmission, which makes it difficult to expand and deploy devices, and results in insufficient signal latency and reliability.
It adopts a distributed memory orthogonal architecture based on CXL, connecting the host node module and the memory switching network module through orthogonal connectors, eliminating the need for an external CXL switch, simplifying signal connections, and improving signal integrity and heat dissipation.
It enables efficient sharing and expansion of memory pool resources, reduces device energy consumption and maintenance costs, and improves system reliability and scalability.
Smart Images
Figure CN117851283B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of cloud service technology, and in particular relates to a distributed memory orthogonal architecture design method and system based on CXL. Background Technology
[0002] CXL is a high-speed serial protocol that allows for fast and reliable data transfer between different components within a computer system. Introduced in 2020, CXL was jointly designed by companies such as Intel, Dell, and HP. It aims to address bottlenecks in high-performance computing, including memory capacity, memory bandwidth, and I / O latency. CXL also enables memory expansion and sharing, and can communicate with peripherals such as computing accelerators (GPUs and FPGAs), providing faster and more flexible data exchange and processing.
[0003] The CXL protocol comprises three sub-protocols:
[0004] CXL.io Protocol: This mode allows memory to be extended to external devices, resulting in faster data transfer speeds. CXL.io connects the CPU and external devices via the PCIe bus, enabling the CPU to share memory with external devices and directly access their I / O resources.
[0005] CXL.cache protocol: This mode improves performance by caching memory on external devices. CXL.cache mode allows the CPU to retain the most frequently used data in its local cache, while storing less frequently used data on external devices. This reduces memory access time and improves overall system performance.
[0006] CXL.memory protocol: This mode allows external devices to be used as main memory, thus enabling larger memory capacity. CXL.memory mode allows the CPU to treat external devices as extended memory, allowing for the storage of more data. This improves system reliability because even in the event of a memory failure, the CPU can still continue using external devices.
[0007] Three types of CXL technology:
[0008] The first type: Accelerator cards or add-on cards installed via PCIe slots. These cards can be integrated with existing systems and communicate directly with the CPU via the CXL interface to provide faster data transfer speeds. They are used for high-speed caching devices such as network cards.
[0009] The second type has all the features of the first type of device and is typically used in scenarios with high-density computing, such as GPU accelerators.
[0010] The third type: a dedicated storage device that communicates directly with the host processor and can use the CXL protocol to achieve low-latency, high-throughput data transfer. It serves as a memory buffer to extend memory bandwidth and capacity.
[0011] Distributed computing is a research direction in computer science. Distributed memory technology benefits from the development of CXL technology. Through the CXL.cache and CXL.memory protocols, a third type of memory pool device is connected to the server through a CXL switch to form a switching network. This solves the problem of resource waste caused by the variable core-to-memory ratio of cloud products on a single cloud server. Memory can be requested or released on demand through unified management software, and it can be horizontally scaled up and down through the CXL switching network, improving the reliability and availability of the entire system.
[0012] Traditional distributed storage solutions typically use CXL switches as the switching network between the host and memory pool devices. While CXL has many advantages, at the physical layer, it is very similar to PCIe and has the same limitations as PCIe: unlike a regular Ethernet interface, PCIe cannot transmit over long distances, and its RX and TX reference clocks must be from the same source; it uses handshake and error retransmission mechanisms to achieve reliable transmission, so its latency cannot be optimized.
[0013] These two limitations prevent it from achieving long-distance transmission. Although it supports multi-level switching cascading, the number of hops must be minimized to ensure low latency. Therefore, in traditional solutions, the host, CXL switch, and memory pool are three separate devices, and the high-speed PCIe bus needs to extend outside the chassis to establish links with external devices. Since the PCIe physical layer lacks optical communication capabilities, external connections can only be made via electrical signals, placing high demands on cable length and link shielding, making expansion or long-distance deployment extremely difficult. Summary of the Invention
[0014] In view of the shortcomings of the existing technology, the purpose of this invention is to provide a distributed memory orthogonal architecture design method and system based on CXL. By adopting an orthogonal architecture solution, complex long-distance signal wiring and vias are reduced, signal connections are simplified, and backplane routing is eliminated. The external CXL switch is removed, significantly shortening the physical link length from the host to memory pool devices and improving signal integrity. Simultaneously, the orthogonal connector system maximizes the chassis's heat dissipation and airflow, further improving signal integrity performance. The mechanical structure design exhibits strong stability and durability.
[0015] In a first aspect, the present invention proposes a distributed memory orthogonal architecture based on CXL, comprising:
[0016] The host node module and the memory switching network module are respectively disposed on a first plate-shaped device and a second plate-shaped device. The first plate-shaped device and the second plate-shaped device are connected by a connector and are orthogonally connected; wherein, in the orthogonal connection, the first plate-shaped device and the second plate-shaped device are perpendicular to each other.
[0017] Furthermore, the host node module includes a first main CPU, a second main CPU, and multiple orthogonal connector sub-terminals. The first main CPU and the second main CPU communicate via UPI. The first main CPU is connected to each of the multiple orthogonal connector sub-terminals, and the second main CPU is connected to each of the multiple orthogonal connector sub-terminals, thereby enabling the information of the first main CPU and the second main CPU to be transmitted through the orthogonal connector sub-terminals.
[0018] Furthermore, the host node module also includes a PCH module, a first low-speed connector, and a first BMC module. The first main CPU communicates with the PCH module through a DMI interface. The PCH module communicates with the first low-speed connector through a first channel and a second channel. The first channel is for the PCH module to communicate with the first low-speed connector through the first BMC module, and the second channel is for the PCH module to communicate directly with the first low-speed connector.
[0019] Furthermore, the communication methods between the PCH module and the first BMC module include eSPI, PCIe, and USB.
[0020] Furthermore, the host node module also includes a first input terminal and a second input terminal, wherein the first input terminal communicates with the first main CPU and the second input terminal communicates with the second main CPU.
[0021] Furthermore, the memory switching network module includes multiple orthogonal connector female terminals, an onboard CXL switching chip, a memory pool, an embedded CPU, a second BMC module, and a second low-speed connector. The input terminal of the onboard CXL switching chip is connected to the multiple orthogonal connector female terminals, and the output terminal of the onboard CXL switching chip is connected to the memory pool. The low-speed connector is connected to the second BMC module and the embedded CPU respectively. The second BMC module communicates with the embedded CPU, and the embedded CPU communicates with the onboard CXL switching chip.
[0022] Furthermore, the memory pool includes multiple memory controllers and multiple memories, with each memory controller and memory connected to the others. The onboard CXL switching chip is connected to each of the multiple memory controllers.
[0023] Furthermore, the memory pool includes multiple CXL memories, and the onboard CXL switching chip is connected to each of the multiple CXL memories.
[0024] Furthermore, the first plate-shaped device is a first PCB board, and the second plate-shaped device is a second PCB board.
[0025] Furthermore, the connector is an orthogonal connector.
[0026] The beneficial effects of this invention are as follows:
[0027] This invention provides a CXL-based distributed memory orthogonal architecture, employing an integrated chassis solution with orthogonal architecture. It consists of a front-mounted horizontal host node, which can be a computing node (CPU or GPU), I / O node, or storage node; and a rear-mounted vertical memory switching network. The front-mounted host nodes and the rear-mounted memory switching network are directly connected via orthogonal connectors. The memory switching network forms a third type of memory switching network through onboard CXL switching chips and CXL DRAM memory or E3.S packaged CXL DRAM disks. Data exchange between the host and the memory pool is completed within the integrated chassis, without external wiring, increasing link reliability. The integrated chassis also facilitates maintenance, and the compact design saves overall energy consumption, thereby reducing TCO. The orthogonal memory switching network can be expanded or reduced as needed, with an overall memory capacity equivalent to a memory pool composed of six CXL switches. Simultaneously, pooled memory or expanded memory pools can be provided externally through CXL ports via I / O modules. Attached Figure Description
[0028] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts. It is obvious that the drawings described below are merely some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings.
[0029] Figure 1 A schematic diagram of a distributed memory orthogonal architecture based on CXL provided in an embodiment of the present invention;
[0030] Figure 2 A physical schematic diagram of a distributed memory orthogonal architecture based on CXL provided for an embodiment of the present invention;
[0031] Figure 3 A schematic diagram of a host node module structure based on a CXL-based distributed memory orthogonal architecture is provided for an embodiment of the present invention.
[0032] Figure 4This is a schematic diagram of the structure of a memory switching network module provided in an embodiment of the present invention;
[0033] Figure 5 This is a schematic diagram of another memory exchange network module provided in an embodiment of the present invention.
[0034] Figure label:
[0035] 1-Host node module; 2-Memory switching network module; 21-Connector; 3-First main CPU; 4-Second main CPU; 5-Orthogonal connector daughter terminal; 6-First BMC module; 7-First low-speed connector; 8-First input terminal; 9-Second input terminal; 10-PCH module; 11-Orthogonal connector female terminal; 12-Onboard CXL switching chip; 13-Memory pool; 14-Embedded CPU; 15-Second BMC module; 16-Second low-speed connector; 17-Memory controller; 18-Memory; 19-CXL memory. Detailed Implementation
[0036] To enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. It should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0037] Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts disclosed in this invention.
[0038] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the system or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. The terms "installed," "connected," and "linked" should be interpreted broadly; for example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal communication of two elements. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.
[0039] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of methods and systems consistent with some aspects of the invention as detailed in the appended claims.
[0040] This invention proposes a distributed memory orthogonal architecture based on CXL. To address the limitations of traditional distributed storage solutions in achieving long-distance transmission, while supporting multi-level switch cascading, the number of hops must be minimized for low latency. In traditional solutions, the host, CXL switch, and memory pool are three separate devices, requiring the high-speed PCIe bus to extend outside the chassis to establish links with external devices. Since the PCIe physical layer lacks optical communication capabilities, external connections can only be made via electrical signals, placing high demands on cable length and link shielding, making expansion or long-distance deployment extremely difficult.
[0041] System Implementation Examples
[0042] Reference manual attached Figure 1 The diagram shows a schematic of a distributed memory orthogonal architecture based on CXL provided by an embodiment of the present invention.
[0043] Reference manual attached Figure 2 The diagram shows a physical schematic of a distributed memory orthogonal architecture based on CXL provided by an embodiment of the present invention.
[0044] The present invention provides a distributed memory orthogonal architecture based on CXL, comprising: a host node module 1 and a memory switching network module 2, wherein the host node module 1 and the memory switching network module 2 are respectively disposed on a first plate-shaped device and a second plate-shaped device, the first plate-shaped device and the second plate-shaped device are connected by a connector 21, and the first plate-shaped device and the second plate-shaped device are orthogonally connected; wherein, in the orthogonal connection, the first plate-shaped device and the second plate-shaped device are perpendicular to each other.
[0045] In one possible implementation, the first plate-shaped device is a first PCB board, and the second plate-shaped device is a second PCB board.
[0046] In one possible implementation, the connector is an orthogonal connector.
[0047] Furthermore, CXL (Compute Express Link) technology is a new type of high-speed interconnect technology designed to provide higher data throughput and lower latency to meet the needs of modern computing and storage systems.
[0048] Furthermore, orthogonal architecture is a type of switch architecture, where orthogonal connections are made using orthogonal connectors. By using orthogonal connectors, true backplane-less switching is achieved, resulting in high transmission rates. This overcomes the line speed bottleneck of traditional backplane architectures and is the mainstream solution for integrated chassis-type switching matrices.
[0049] In one possible implementation, the host node module 1 includes a first main CPU 3, a second main CPU 4, and a plurality of orthogonal connector sub-terminals 5. The first main CPU 3 and the second main CPU 4 communicate via UPI. The first main CPU 3 is connected to the plurality of orthogonal connector sub-terminals 5, and the second main CPU 4 is connected to the plurality of orthogonal connector sub-terminals 5, so that information of the first main CPU 3 and the second main CPU 4 is transmitted through the orthogonal connector sub-terminals 5.
[0050] Furthermore, UPI, Intel's Ultra Path Interconnect, replaces QPI technology, offering higher communication speeds, higher efficiency, and lower power consumption.
[0051] In one possible implementation, the host node module further includes a PCH module 10, a first low-speed connector 7, and a first BMC module 6. The first main CPU 3 communicates with the PCH module 10 through a DMI interface. The PCH module 10 and the first low-speed connector 7 communicate through a first channel and a second channel. The first channel is for the PCH module 10 to communicate with the first low-speed connector 7 through the first BMC module 6, and the second channel is for the PCH module 10 to communicate directly with the first low-speed connector 7.
[0052] Furthermore, the DMI interface carries two main types of information transmission: control information and data transmission information. Control information includes information about hardware components and communication rules between the processor and chipset. Additionally, the System Management Bus (SMBus) and Low Pin Count (LPC) bus are also connected to the CPU via the DMI. In this way, the DMI can control multiple peripherals and transmit relevant information. Meanwhile, expansion cards connected to the DMI interface can typically share signals with the PCH. DMI, short for Direct Media Interface, is Intel's interface standard for connecting the motherboard and CPU. It is designed as a high-bandwidth, low-latency interface for transmitting data, raw commands, and control information.
[0053] BMC circuits are a new type of integrated circuit that can integrate multiple circuit elements and components into a single package, thereby saving space and reducing the size of the circuit board. It works by laminating multiple layers of thermoplastic substrates to form a package of multiple circuit elements and components.
[0054] In one possible implementation, the communication methods between the PCH module 10 and the first BMC module 6 include eSPI, PCIe, and USB.
[0055] In one possible implementation, the host node module 1 further includes a first input terminal 8 and a second input terminal 9. The first input terminal 8 communicates with the first main CPU 3, and the second input terminal 9 communicates with the second main CPU 4. The first input terminal and the second input terminal are respectively used to connect to the front panel to realize information communication.
[0056] In one possible implementation, the memory switching network module 2 includes multiple orthogonal connector female terminals 11, an onboard CXL switching chip 12, a memory pool 13, an embedded CPU 14, a second BMC module 15, and a second low-speed connector 16. The input terminal of the onboard CXL switching chip 12 is connected to the multiple orthogonal connector female terminals 11, and the output terminal of the onboard CXL switching chip 12 is connected to the memory pool 13. The second low-speed connector 16 is connected to the second BMC module 15 and the embedded CPU 14, respectively. The second BMC module 15 communicates with the embedded CPU 14, and the embedded CPU 14 communicates with the onboard CXL switching chip 12.
[0057] Among them, multiple orthogonal connector females communicate with the first main CPU and the second main CPU through PCIe 6.0*8 connectors, and the first main CPU communicates with the first input terminal and the second main CPU communicates with the second input terminal through PCIe 6.0*16.
[0058] Currently, most mainstream high-speed connector manufacturers offer relevant products, such as Amphenol, TE Connectivity, and Molex. Below are some comparative data examples. PCIe 5.0 uses 32G NRZ signals, requiring connectors that support signal rates above 32G, typically using 85Ω impedance matching. PCIe 6.0 uses 64G PAM4 signals, so connectors supporting PAM4 signals above 64G must be selected, also typically using 85Ω impedance matching. The number of signal pairs is selected based on actual requirements.
[0059]
[0060] In one possible implementation, the memory pool includes a plurality of memory controllers 17 and a plurality of memories 18, the memory controllers 17 and the memories 18 being connected one by one, and the onboard CXL switching chip 12 being connected to the plurality of memory controllers 17 respectively.
[0061] Furthermore, the memory is CXL.DRAM. The memory switching network forms a third type of memory switching network through the onboard CXL switching chip and the CXL DRAM disk.
[0062] In one possible implementation, the memory pool includes a plurality of CXL memories 19, and the onboard CXL switching chip 12 is connected to the plurality of CXL memories 19 respectively.
[0063] Furthermore, the memory is an E3.S packaged CXL DRAM. The memory switching network forms a third type of memory switching network through the onboard CXL switching chip and the E3.S packaged CXL DRAM disk.
[0064] Among them, multiple orthogonal connector females communicate with the onboard CXL switching chip via PCIe 6.0*16 connectors, and the onboard CXL switching chip communicates with multiple memory controllers via PCIe 6.0*4.
[0065] Currently, most mainstream high-speed connector manufacturers offer relevant products, such as Amphenol, TE Connectivity, and Molex. Below are some comparative data examples. PCIe 5.0 uses 32G NRZ signals, requiring connectors that support signal rates above 32G, typically using 85Ω impedance matching. PCIe 6.0 uses 64G PAM4 signals, so connectors supporting PAM4 signals above 64G must be selected, also typically using 85Ω impedance matching. The number of signal pairs is selected based on actual requirements.
[0066]
[0067] This solution can be applied to scenarios such as memory-enhanced virtual machine (VM) applications, where the core-to-memory ratio of a VM is greater than 1:8. Since the cost per MB of DDR is higher than that of CXL DRAM, and servers commonly use DDR memory specifications of 1T-2T, while CXL DRAM can usually reach more than 6T, this invention can define cloud products with a larger core-to-memory ratio. The cloud resource pool memory pooling method breaks the memory limitations of individual servers, allowing computing nodes to share memory capacity, maximizing the saving of single-machine CPU and memory resources, while the integrated power supply and heat dissipation solution improves the overall TCO.
[0068] The beneficial effects of this invention are as follows:
[0069] This invention provides a CXL-based distributed memory orthogonal architecture, employing an integrated chassis solution with orthogonal architecture. It consists of a front-mounted horizontal host node, which can be a computing node (CPU or GPU), I / O node, or storage node; and a rear-mounted vertical memory switching network. The front-mounted host nodes and the rear-mounted memory switching network are directly connected via orthogonal connectors. The memory switching network forms a third type of memory switching network through onboard CXL switching chips and CXL DRAM memory or E3.S packaged CXL DRAM disks. Data exchange between the host and the memory pool is completed within the integrated chassis, without external wiring, increasing link reliability. The integrated chassis also facilitates maintenance, and the compact design saves overall energy consumption, thereby reducing TCO. The orthogonal memory switching network can be expanded or reduced as needed, with an overall memory capacity equivalent to a memory pool composed of six CXL switches. Simultaneously, pooled memory or expanded memory pools can be provided externally through CXL ports via I / O modules.
[0070] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention, and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the protection scope of the present invention.
Claims
1. A distributed memory orthogonal architecture based on CXL, characterized in that, include: The host node module and the memory switching network module are respectively disposed on a first plate-shaped device and a second plate-shaped device. The first plate-shaped device and the second plate-shaped device are connected by a connector and are orthogonally connected. In the orthogonal connection, the first plate-shaped device and the second plate-shaped device are perpendicular to each other. The memory switching network module includes multiple orthogonal connector female terminals, an onboard CXL switching chip, a memory pool, an embedded CPU, a second BMC module, and a second low-speed connector. The input terminal of the onboard CXL switching chip is connected to the multiple orthogonal connector female terminals, and the output terminal of the onboard CXL switching chip is connected to the memory pool. The second low-speed connector is connected to both the second BMC module and the embedded CPU. The second BMC module communicates with the embedded CPU, and the embedded CPU communicates with the onboard CXL switching chip.
2. The distributed memory orthogonal architecture according to claim 1, characterized in that, The host node module includes a first main CPU, a second main CPU, and multiple orthogonal connector sub-terminals. The first main CPU and the second main CPU communicate via UPI. The first main CPU is connected to each of the multiple orthogonal connector sub-terminals, and the second main CPU is connected to each of the multiple orthogonal connector sub-terminals, thereby enabling the transmission of information between the first main CPU and the second main CPU through the orthogonal connector sub-terminals.
3. The distributed memory orthogonal architecture according to claim 2, characterized in that, The host node module further includes a PCH module, a first low-speed connector, and a first BMC module. The first main CPU communicates with the PCH module through a DMI interface. The PCH module communicates with the first low-speed connector through a first channel and a second channel. The first channel is for the PCH module to communicate with the first low-speed connector through the first BMC module, and the second channel is for the PCH module to communicate directly with the first low-speed connector.
4. The distributed memory orthogonal architecture according to claim 3, characterized in that, The communication methods between the PCH module and the first BMC module include eSPI, PCIe, and USB.
5. The distributed memory orthogonal architecture according to claim 2, characterized in that, The host node module further includes a first input terminal and a second input terminal, wherein the first input terminal communicates with the first main CPU and the second input terminal communicates with the second main CPU.
6. The distributed memory orthogonal architecture according to claim 1, characterized in that, The memory pool includes multiple memory controllers and multiple memories, with each memory controller and memory connected to the others. The onboard CXL switching chip is connected to each of the multiple memory controllers.
7. The distributed memory orthogonal architecture according to claim 1, characterized in that, The memory pool includes multiple CXL memories, and the onboard CXL switching chip is connected to each of the multiple CXL memories.
8. The distributed memory orthogonal architecture according to claim 1, characterized in that, The first plate-shaped device is a first PCB board, and the second plate-shaped device is a second PCB board.
9. The distributed memory orthogonal architecture according to claim 1, characterized in that, The connector is an orthogonal connector.