CPU-fpga heterogeneous memory pooling extension system
Patent Information
- Application Number
- CN202522353896.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Utility models(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-06
- Publication Date
- 2026-10-23
- Estimated Expiration
- 2035-11-06
AI Technical Summary
[0004]本实用新型的目的在于提供一种基于CPU-FPGA异构内存池化扩展系统，以解决现有异构系统中远端内存访问的延迟高、CPU占用率高、带宽资源受限和系统扩展性不足的问题
[0023]与现有技术相比，本实用新型的有益效果是：通过零拷贝机制实现纳秒级响应，NOC架构与协议优化突破传输带宽瓶颈、实现动态流量调节与无阻塞数据通路，互联架构实现互联和内存容量扩展，提升了数据中心的高吞吐、低延迟处理能力，为解决异构计算环境中的内存瓶颈问题提供了高效解决方案。
Smart Images
Figure CN224889149U_ABST
Abstract
Description
Technical Field
[0001] This utility model relates to the field of data storage and computer technology, specifically to a CPU-FPGA heterogeneous memory pooling expansion system. Background Technology
[0002] In heterogeneous computing architectures, the collaborative work of CPUs and FPGAs is the core mode for improving data processing efficiency. By leveraging the general computing power of CPUs and the parallel acceleration characteristics of FPGAs, it is widely used in high-performance scenarios such as data centers and edge computing. However, memory access bottlenecks remain a key factor restricting breakthroughs in system performance, especially with increasingly stringent requirements for high throughput and low latency, where the limitations of traditional technical solutions are becoming more and more apparent.
[0003] Traditional data transmission and memory architectures suffer from multi-dimensional performance bottlenecks. While XDMA-based transmission can establish data pathways, its design is better suited for scenarios with few transmissions, large batches, and large data units. Its efficiency drops significantly in small-unit, large-batch transmissions, failing to meet the demands of fine-grained data scheduling. Shared memory architectures face three core challenges: First, the non-zero-copy transmission mechanism leads to frequent data movement operations consuming CPU resources, weakening the effective utilization of computing resources. Second, the capacity limitation of the memory controller; in traditional computing architectures, the memory capacity supported by a single CPU is limited by the physical design of the memory controller and the address bus width, making it difficult to meet the terabyte-level memory space requirements of large-scale data processing. Third, the lack of efficient flow control mechanisms results in large fluctuations in data transmission latency, especially prone to data congestion under peak loads. Utility Model Content
[0004] The purpose of this invention is to provide a CPU-FPGA heterogeneous memory pooling expansion system to solve the problems of high latency, high CPU utilization, limited bandwidth resources, and insufficient system scalability in existing heterogeneous systems for remote memory access.
[0005] To achieve the above objectives, this utility model provides the following technical solution: The system includes: a software layer, a conversion interconnection logic layer, and a memory expansion logic layer;
[0006] The software layer includes a CPU and a matching motherboard, used for the operation of the operating system on the x86 platform, loading of the QDMA driver, and data access to the DDR memory of the FPGA platform based on the PCIe protocol.
[0007] The conversion interconnection logic layer comprises three parts: QDMA, address interconnection mapping, and protocol conversion. It is responsible for establishing PCIe links with external host systems, receiving memory access requests from hosts, completing the conversion from PCIe protocol to Aurora protocol and address mapping, and realizing unified addressing of host address space and extended memory space.
[0008] More preferably, the memory extension logic layer includes a protocol conversion and NOC section, which receives access requests via the Aurora protocol and converts them into physical access operations for DDR memory, thereby realizing read and write control of the extended memory.
[0009] More preferably, in the software layer, the CPU motherboard provides power supply, memory, BIOS and other hardware for the whole machine to run stably, load the QDMA driver based on the PCIe protocol, access address range 0x00_0000_0000-0xFF_FFFF_FFFF, and the physical SERDEs connected to the logical layer are gold fingers, which support a maximum of Gen4x8.
[0010] More preferably, the conversion interconnection logic layer is based on the Ultra FPGA platform. The QDMA part is responsible for establishing a PCIe link with the external host system and receiving the host's memory access request. The address mapping part realizes the mapping of the DDR memory address of each memory expansion card by QDMA. The protocol conversion part completes the conversion from PCIe protocol to Aurora protocol, realizing the unified addressing of the host address space and the extended memory space.
[0011] More preferably, the memory expansion logic layer consists of four Versal series FPGA memory expansion cards with high logic density and multiple hard cores to achieve faster response speed and higher data transmission rate. Each memory expansion card provides power supply, clock, dual parallel mode FLASH configuration, PCIe link status and DDR verification status indicator for stable FPGA operation. The protocol conversion part of this layer realizes data interaction of Aurora protocol and AXI bus type conversion. The slave end accesses the NOC part. The NOC part realizes functions such as bus cross-clock, data flow control, read and write priority, DDR parameter matching, DDR verification initialization and DDR driver to realize high bandwidth and large capacity data access of DDR.
[0012] Furthermore, the system startup initialization steps are as follows:
[0013] Step 1: Power on the FPGA platform that connects to the interconnect logic layer, complete the FPGA power-on configuration in QSPIx8 FLASH mode, and input the clock and reset. After the configuration is completed, the completion indicator light on the FPGA board platform will light up.
[0014] Step 2: The QDMA portion of the interconnect logic layer is used as an EP device for PCIe, and registers and state machine are initialized simultaneously;
[0015] Step 3: The memory expansion logic layer provides secondary power through the gold fingers on the interconnect logic layer board to initialize the protocol conversion logic. At the same time, the NOC part completes the initialization and self-verification of DDR. After the DDR verification is successful, the DDR loading success indicator light on the FPGA board platform of the memory expansion logic layer lights up.
[0016] Step 4: Power on the software layer. First, power on the CPU and its supporting boards. As a PCIe RC device, the CPU performs PCIe initialization work such as scanning, enumerating, and establishing links on the FPGA device, thus completing the basic PCIe initialization work. At this point, all initialization work is complete.
[0017] Further optimized, the read / write steps of this system are as follows:
[0018] Step 1: The CPU part of the software layer loads the QDMA driver, inputs read and write commands to enter the driver, the driver converts the corresponding instructions into descriptors, and transmits credit interaction and data transmission requests to the interconnection logic layer through the board's gold fingers' serdes.
[0019] Step 2: The interconnect logic layer receives requests transmitted to the FPGA platform via the driver, performs descriptor, application, and data interaction, and realizes the data transmission of PCIe to Aurora protocol conversion through the AXI bus interconnect, mapping, and protocol conversion parts.
[0020] Step 3: The protocol conversion part of the memory extension logic layer receives the read and write requests from the upper layer and sends the bus data request to the NOC part. Internally, it performs bus cross-clocking, data flow control, read and write priority, physical layer instruction conversion matching the DDR model, and hard core acceleration processing of each instruction and data to drive the DDR efficiently. Finally, the bus write request completes the data writing to the DDR and completes the response return, and the bus read request completes the data reading and return.
[0021] Alternatively, the system's software layer can be replaced with servers or hosts on other x86 platforms, requiring the software layer hardware to provide a gold finger slot and the software to install the QDMA software driver, thus ensuring compatibility with this system for data access.
[0022] Furthermore, the NOC portion of the system's memory expansion card is compatible with 16GB, 32GB, and 64GB DIMM memory modules, allowing for the insertion of up to four DIMM modules.
[0023] Compared with existing technologies, the beneficial effects of this utility model are: achieving nanosecond-level response through a zero-copy mechanism, breaking through the transmission bandwidth bottleneck through NOC architecture and protocol optimization, realizing dynamic traffic adjustment and non-blocking data paths, and achieving interconnection and memory capacity expansion through interconnection architecture, thereby improving the high throughput and low latency processing capabilities of data centers and providing an efficient solution to the memory bottleneck problem in heterogeneous computing environments. Attached Figure Description
[0024] Figure 1 This is a structural diagram of the CPU-FPGA heterogeneous memory pooling expansion system of this utility model;
[0025] Figure 2 This is a structural diagram of the interconnection logic layer of this utility model;
[0026] Figure 3 This is a diagram of the memory extension logic layer structure of this utility model;
[0027] Figure 4 This is a system initialization flowchart of this utility model;
[0028] Figure 5 This is a flowchart of the system operation read and write process of this utility model;
[0029] Figure 6 This is a system compatibility extension diagram of this utility model. Detailed Implementation
[0030] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention.
[0031] Example
[0032] Please see Figure 1-6 This invention provides a technical solution: by heterogeneous memory pooling, DDR memory resources scattered across memory expansion cards are abstracted into a unified memory pool, with global address management and access scheduling performed by the logic platform of the interconnect layer. Utilizing protocol conversion and distributed interconnection technologies, memory resource pooling expansion under the heterogeneous architecture of CPU and FPGA at the software layer is achieved, breaking through the physical capacity limitations of traditional memory architectures.
[0033] Each module forms a closed loop through a clearly defined data flow path: host access requests enter the FPGA via the PCIe interface, after protocol and address conversion is completed, they are routed to the target memory expansion card through the distributed interconnect layer, and finally the memory expansion card performs DDR memory access and returns the result. The entire process ensures data consistency and transmission efficiency through a cross-clock domain synchronization mechanism.
[0034] The system structure is as follows: Figure 1As shown, this system includes: a software layer, a conversion and interconnection logic layer, and a memory expansion logic layer;
[0035] Software layer: Based on the x86 platform, including CPU and matching motherboard, used for operating system operation on the x86 platform, QDMA driver loading, and data access to DDR memory to the FPGA platform based on PCIe protocol;
[0036] The CPU is an AMD R7 processor. The motherboard provides power, memory, BIOS and other hardware for the entire system to run stably. It loads the QDMA driver based on the PCIe protocol, with an access address range of 0x00_0000_0000-0xFF_FFFF_FFFF. The physical SERDEs connected to the logical layer are gold fingers, which support a maximum of Gen4x8.
[0037] like Figure 2 As shown, the interconnect logic layer is implemented based on the Xilinx ultra series FPGA platform and includes three parts: QDMA, address interconnect mapping, and protocol conversion. The QDMA part is responsible for establishing a PCIe link with the external host system and receiving the host's memory access requests. The address mapping part implements the mapping of DDR memory addresses of each memory expansion card by QDMA. The protocol conversion part completes the conversion from PCIe protocol to Aurora protocol, realizing unified addressing between the host address space and the extended memory space.
[0038] The QDMA section of the interconnect logic layer has an input reference clock of 100MHz and an output bus clock of 250MHz. The high-speed transceiver of the PHY layer is GTY. The data link and transaction layers use queue direct storage IP QDMA, which supports up to 512 queues. There is also an AXI bus interface as the master terminal to access the FPGA to realize multi-queue parallel data transmission based on the PCIe protocol.
[0039] The address interconnection mapping part of the interconnection logic layer will implement a master-slave interconnection structure by configuring the smartinterconnect IP, and will evenly map the QDMA driver address range to the memory expansion card on each SLOT.
[0040] The protocol conversion section of the conversion interconnect logic layer uses asynchronous FIFO to implement data buffering and cross-clock processing from the QDMA 250MHz clock domain to the Aurora 390MHz clock domain. It also converts the AXI_FULL bus protocol into the AXI_STREAM bus mode through logic design and inputs it into the 25Gb x8 configured Aurora IP to realize the function of transferring data from the conversion interconnect logic layer to the memory extension logic layer.
[0041] like Figure 3As shown, the memory extension logic layer includes protocol conversion and NOC parts. It receives access requests through the Aurora protocol and converts them into physical access operations for DDR memory, thereby realizing read and write control of the extended memory.
[0042] This layer is based on the Xilinx Versal series FPGA board platform. In addition, this model of FPGA platform has high logic density and multiple hard cores to achieve faster response speed and higher data transmission rate. The board provides status indicators for stable FPGA operation, including power supply, clock, FLASH configuration, PCIe link status and DDR verification. The FPGA configuration device adopts a dual parallel mode QSPIx8 port FLASH to improve the FPGA boot configuration speed.
[0043] The protocol conversion part of the memory extension logic layer: Aurora IP with 25Gb 8lane is used to implement the data interaction of Aurora protocol, and the AXI_STREAM bus protocol is converted into the master end of the AXI_FULL bus protocol through logic design, and the slave end is connected to the NOC part;
[0044] The NOC section of the memory expansion logic layer includes an input system and a 200MHz reference multiplexed clock. It utilizes NOC hard-core IP to achieve hard-core acceleration while also implementing functions such as bus clock crossing, data flow control, read / write priority, DDR parameter matching, DDR initialization verification, and DDR driver functionality. It supports four external 64GB DIMM modules, providing a total DDR capacity of 256GB per memory expansion card and a total DDR capacity of 1TB for the memory expansion logic layer, enabling high-bandwidth, high-capacity data access from DDR.
[0045] The system startup and initialization steps are as follows: Figure 4 As shown, the specific steps are as follows:
[0046] Step 1: Power on the FPGA platform that connects to the interconnect logic layer, complete the FPGA power-on configuration in QSPIx8 FLASH mode, and input the clock and reset. After the configuration is completed, the completion indicator light on the FPGA board platform will light up.
[0047] Step 2: The QDMA portion of the interconnect logic layer is used as an EP device for PCIe, and registers and state machine are initialized simultaneously;
[0048] Step 3: The memory expansion logic layer provides secondary power through the gold fingers on the interconnect logic layer board to initialize the protocol conversion logic. At the same time, the NOC part completes the initialization and self-verification of DDR. After the DDR verification is successful, the DDR loading success indicator light on the FPGA board platform of the memory expansion logic layer lights up.
[0049] Step 4: Power on the software layer. First, power on the CPU and its supporting boards. As a PCIe RC device, the CPU performs PCIe initialization work such as scanning, enumerating, and establishing links on the FPGA device, thus completing the basic PCIe initialization work. At this point, all initialization work is complete.
[0050] The basic read / write steps of this system are as follows: Figure 5 As shown, the steps are as follows:
[0051] Step 1: The CPU part of the software layer loads the QDMA driver, inputs read and write commands to enter the driver, the driver converts the corresponding instructions into descriptors, and transmits credit interaction and data transmission requests to the interconnection logic layer through the board's gold fingers' serdes.
[0052] Step 2: The interconnect logic layer receives requests transmitted to the FPGA platform via the driver, performs descriptor, application, and data interaction, and realizes the data transmission of PCIe to Aurora protocol conversion through the AXI bus interconnect, mapping, and protocol conversion parts.
[0053] Step 3: The protocol conversion part of the memory extension logic layer receives the read and write requests from the upper layer and sends the bus data request to the NOC part. Internally, it performs bus cross-clocking, data flow control, read and write priority, physical layer instruction conversion matching the DDR model, and hard core acceleration processing of each instruction and data to drive the DDR efficiently. Finally, the bus write request completes the data writing to the DDR and completes the response return, and the bus read request completes the data reading and return.
[0054] This system's compatibility and extensions are described as follows: Figure 6 As shown, the software layer can also be replaced with other X86 platform servers or hosts. The software layer hardware must provide a gold finger slot and the software must be able to install the QDMA software driver to be compatible with this system for data access. The memory expansion logic layer is compatible with 16GB, 32GB, and 64GB DIMM memory modules, and a maximum of four DIMM modules can be inserted.
[0055] Although embodiments of the present invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the present invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A CPU-FPGA heterogeneous memory pooling expansion system, comprising a software layer, a conversion interconnection logic layer, and a memory expansion logic layer, characterized in that: The software layer is connected to the conversion interconnection logic layer, and the conversion interconnection logic layer is connected to the memory expansion logic layer; The software layer includes a CPU and a matching motherboard, which is used for operating system operation, QDMA driver loading, and data access to DDR memory to the FPGA platform based on the PCIe protocol. The conversion interconnection logic layer includes three parts: QDMA, address interconnection mapping, and protocol conversion. It is used to establish a PCIe link with the external host system and receive the host's memory access requests. The memory extension logic layer, including the protocol conversion and NOC parts, receives access requests via the Aurora protocol and converts them into physical access operations for DDR memory.
2. The CPU-FPGA heterogeneous memory pooling expansion system according to claim 1, characterized in that: The software layer is based on the x86 platform. The motherboard provides power, memory, and BIOS hardware for the entire system. The physical SERDEs that connect to the logic layer are the gold fingers.
3. The CPU-FPGA heterogeneous memory pooling expansion system according to claim 1, characterized in that: The conversion interconnect logic layer is based on the Ultra FPGA platform. QDMA establishes a PCIe link with the external host system and receives memory access requests from the host.
4. The CPU-FPGA heterogeneous memory pooling expansion system according to claim 3, characterized in that: The aforementioned conversion interconnection logic layer includes an address mapping section that performs QDMA mapping of DDR memory addresses for each memory expansion card, and a protocol conversion section that completes the conversion from PCIe protocol to Aurora protocol.
5. The CPU-FPGA heterogeneous memory pooling expansion system according to claim 1, characterized in that: The memory expansion logic layer consists of four Versal series FPGA memory expansion cards with high logic density and multiple hard cores.
6. The CPU-FPGA heterogeneous memory pooling expansion system according to claim 5, characterized in that: The memory expansion logic layer provides each memory expansion board with power supply, clock, dual parallel mode FLASH configuration, PCIe link status and DDR verification status indicator for stable FPGA operation. The protocol conversion part performs data interaction of Aurora protocol and AXI bus type conversion. The slave end accesses the NOC part, which implements bus cross-clock, data flow control, read and write priority, DDR parameter matching, DDR verification initialization and DDR driver functions.
7. The CPU-FPGA heterogeneous memory pooling expansion system according to claim 6, characterized in that: The NOC section of the memory expansion card is compatible with 16GB, 32GB, and 64GB DIMM memory modules.