GPGPU (General Purpose Graphics Processing Unit) video memory expansion method and system based on high-speed interconnection bus

Abstract

The invention discloses a GPGPU (General Purpose Graphics Processing Unit) video memory extension system and method based on a high-speed interconnection bus. According to the system, an independent high-capacity memory pool is constructed outside a single GPU, and a GPU high-speed interconnection bus is utilized, so that transparent expansion of a GPU video memory is realized. The system comprises a GPU side high-speed interconnection interface module, an external memory controller module and an external memory pool. Wherein the interface module adopts a high-speed interconnection protocol and provides bandwidth with the same magnitude as that of an onboard video memory; the external controller is responsible for translating a bus transaction into a standard DDR command to realize efficient management of mass general memories; driving software constructs a uniform video memory address space, and supports page-level automatic migration by taking an onboard video memory as a hot layer and taking an external memory as a warm layer through a hierarchical memory management mechanism. The problem that the capacity of a single GPU video memory is limited is solved, the video memory can be expanded to the TB level, the system cost is remarkably reduced while high performance is kept, and the method has important application value.

Description

Technical Field

[0001] This invention relates to the field of computer hardware and architecture technology, specifically to memory management and expansion technology for deep learning processors. Background Technology

[0002] In recent years, with the development of deep learning, especially large language models (LLM) and multimodal models, and the increasing demands for computational accuracy and scale in scientific computing, general-purpose graphics processing units (GPGPUs) have become the mainstream high-performance computing accelerators. However, there is a serious mismatch between the growth rate of GPGPU computing power and the growth rate of its onboard memory capacity. Cutting-edge AI models (such as models with trillions of parameters) often require hundreds of GB or even TB of memory, far exceeding the physical memory capacity provided by any single flagship GPU currently on the market (typically 24GB to 96GB). This mismatch constitutes the so-called "memory wall" problem, which greatly limits the execution efficiency and feasibility of large-scale computing tasks.

[0003] To alleviate this problem, existing technologies mainly employ the following solutions:

[0004] Option 1: Multi-GPU Parallel Computation and Model Splitting

[0005] This is currently the most mainstream solution in the industry, connecting multiple GPUs to form a computing cluster through high-speed interconnect technologies such as NVIDIA's NVLink or AMD's Infinity Fabric. Applications (such as deep learning frameworks) use model parallelism techniques, such as tensor parallelism and pipeline parallelism, to distribute a huge model across the memory of multiple GPUs. For example, a model with 175 billion parameters can be distributed across eight GPUs with 80GB of memory each.

[0006] Evaluation: While effective, this approach has significant drawbacks. First, it is extremely costly, requiring the purchase of multiple flagship GPUs and motherboards or servers supporting high-speed interconnects. Second, the programming and debugging complexity is high, necessitating intrusive modifications to the model to achieve parallelization and handling complex cross-GPU communication synchronization issues. This not only increases development difficulty but also introduces additional communication overhead, impacting overall computational efficiency. Finally, its scalability remains limited by the total video memory of all GPUs in the cluster, and the granularity of expansion is relatively large.

[0007] Option 2: Expand using system main memory (CPU memory)

[0008] This solution utilizes the PCIe (Peripheral Component Interconnect Express) bus, allowing the GPU to "swap" some data to the much larger system main memory when its own video memory is insufficient. NVIDIA's CUDA Unified Memory technology partially automates this process at the hardware and driver levels, allowing programmers to program in a unified virtual address space, with the system migrating data between GPU video memory and main memory on demand.

[0009] Evaluation: The main problem with this approach lies in its performance bottleneck. The bandwidth of the PCIe bus (even the latest PCIe 5.0, with a bidirectional bandwidth of approximately 128GB / s) and latency (typically in the microsecond range) are one to two orders of magnitude greater than the GPU's onboard memory (such as HBM3, with bandwidth exceeding 3TB / s and latency in the nanosecond range). When computational tasks frequently access data stored in main memory, this high latency and low bandwidth can cause severe stalling of the GPU's computing cores, leading to a sharp decline in computational performance and even significantly diminishing the benefits of expansion. Therefore, this approach is only suitable for specific applications where data access locality requirements are not high or latency is not a major concern.

[0010] Option 3: The new CXL (Compute Express Link) technology

[0011] CXL is an open interconnect protocol built on the PCIe physical layer, designed to enable efficient connections between CPUs, memory, and accelerators. CXL 2.0 and later versions support memory pooling, which theoretically allows accelerators such as GPUs to access an external pool of CXL memory devices via the CXL.mem protocol.

[0012] Assessment: CXL offers a promising standardized path for video memory expansion. However, this technology is still in its early stages of development and adoption. First, mainstream consumer and workstation GPUs do not yet natively and widely support CXL memory expansion. Second, the CXL protocol itself incurs some overhead in its protocol stack for the sake of versatility, and its optimization for high-intensity, fine-grained memory access scenarios like those of GPUs remains to be verified. Whether its actual latency and throughput can meet the needs of GPGPUs remains a key focus of current research.

[0013] In summary, the existing technical solutions have one or more of the following core problems:

[0014] 1) Cost and complexity issues: Multi-GPU solutions have high hardware costs and complex software development.

[0015] Performance bottleneck: PCIe-based system main memory expansion solutions suffer from high bus latency and low bandwidth, resulting in a severe drop in GPU performance.

[0016] 2) Specialization and efficiency issues: Although standardized solutions such as CXL are promising, they are not tailored to the unique memory access mode of GPGPU (high concurrency, strong address randomness, frequent read and write). Their general-purpose protocols may bring unnecessary overhead and cannot achieve the performance level of "onboard memory".

[0017] 3) Coupling Issues: The GPU's video memory capacity is tightly coupled to the computing core on the same PCB board. Users cannot upgrade the video memory capacity independently, on demand, and at low cost, unlike upgrading RAM modules.

[0018] Therefore, there is an urgent need to study a high-performance, low-cost, and easy-to-deploy GPGPU memory expansion system and its supporting methods. This system can provide a single GPGPU with a massive amount of memory space, at the terabyte level, far exceeding its onboard physical memory capacity, while keeping the latency and bandwidth loss of accessing this external memory within an acceptable range, thereby effectively breaking the constraints of the "memory wall". Summary of the Invention

[0019] To address the problems of limited GPGPU memory capacity, high expansion costs, and significant performance degradation after expansion in existing technologies, this invention provides a high-performance, low-cost, and easy-to-deploy GPGPU memory expansion system and its supporting method. This system can provide a single GPGPU with massive memory space, often exceeding its onboard physical memory capacity, at the terabyte level, while keeping the latency and bandwidth loss of accessing this external memory within an acceptable range, thereby effectively breaking the constraints of the "memory wall."

[0020] In a first aspect, embodiments of this application provide a GPGPU memory expansion system based on a high-speed interconnect bus, the system comprising: a GPU chip, a memory control chip for an external memory pool; and

[0021] High-speed interconnect bus: used to connect the GPU chip and the memory controller chip;

[0022] Interconnect interface modules: Located on the GPU chip side and the memory controller chip side respectively, they are connected through a high-speed interconnect bus. The interface modules adopt a high-speed interconnect protocol to provide bandwidth on the same level as the onboard video memory.

[0023] Memory control module: Located inside the memory control chip, it connects to the interconnect interface module on the memory control chip side and connects to the external memory pool. It is used to translate memory requests in bus transactions and manage the external memory pool.

[0024] Driver module: Located within the GPU chip, it is used to create a memory management model, which separates the GPU's onboard video memory and external memory pool, and maps them uniformly into the GPU's virtual address space.

[0025] In this embodiment of the invention, the high-speed interconnect bus adopts a high-speed interconnect protocol, which is divided into a physical layer and a logical protocol layer. The logical protocol layer is used to perform lightweight memory read / write / atomic operation transactions.

[0026] In this embodiment of the invention, the interconnection interface module includes:

[0027] Protocol processing engine module: Used to encapsulate memory access requests from the GPU into lightweight data packets, and unpack the data packets received from the bus and send them to the memory control module for use;

[0028] Physical layer control module: Used to manage signal encoding and decoding, clock data recovery, and link establishment and maintenance.

[0029] In this embodiment of the invention, the interconnect interface module includes: interfaces respectively disposed on the GPU side, which reuse the high-speed bus inside the GPU on the GPU side and are connected to the GPU's cache and memory controller; and

[0030] The interface located on the external memory pool side is connected to the external memory pool side via a new bus protocol for communication with the outside world.

[0031] In this embodiment of the invention, the memory control module communicates with the high-speed interconnect bus through the interface on the external memory pool side, translates the memory transaction requests transmitted on the bus into standard memory commands, and manages the address mapping, refresh, leveling and error correction operations within the memory pool.

[0032] In this embodiment of the invention, the aforementioned driver module creates a hierarchical memory management model, treating the GPU's onboard video memory as hot data and the external memory pool as warm data. The hot data and warm data are mapped to the GPU's virtual address space, and page-level automatic migration can be performed between the two levels according to the frequency and pattern of data access.

[0033] Secondly, embodiments of this application provide a GPGPU memory expansion method based on a high-speed interconnect bus, the method comprising:

[0034] Initialization steps: Initialize the interconnect interface module on the GPU side, and establish a link with the external memory controller chip through the high-speed interconnect bus; initialize the interconnect interface module and memory controller module on the memory controller chip side.

[0035] Driver loading and memory space mapping steps: The operating system loads the driver module, and the driver module modifies the page table structure of the GPU's memory management unit at the kernel level, mapping the physical address space continuously to the high address segment of the GPU's virtual address space. The virtual address space is the video memory address space, which includes the onboard hot memory layer and the external warm memory layer.

[0036] Application memory allocation request steps: When the driver module receives an external memory request, it determines that the memory request space exceeds the GPU's onboard video memory. Then, the driver module allocates a virtual address and the corresponding physical page for the memory request from the address space mapped to the external memory layer.

[0037] Data access steps: The GPU's computing module issues an operation request for a virtual address. The GPU performs corresponding address translation, mapping it to an external warm memory layer. The operation request is encapsulated by the GPU-side interconnect interface module and sent to the interconnect bus. The interconnect interface module on the memory controller chip side receives the operation request and passes it to the memory controller module for reading processing. The external memory pool returns the data to the memory controller module, which encapsulates the data and sends it back to the GPU-side high-speed interconnect interface through the interconnect interface on the memory controller chip side and the interconnect bus. The GPU-side interconnect interface receives the response data packet, unpacks it, and submits the data to the GPU's computing module that issued the request.

[0038] In this embodiment of the invention, the above-mentioned GPGPU memory expansion method based on a high-speed interconnect bus further includes:

[0039] Intelligent data hierarchical migration steps: The driver module monitors the access frequency of data pages located in the warm memory layer through a data access counter; if the access frequency of a certain data page exceeds a preset threshold, the driver module initiates a direct memory access operation, finds a free or cold data page in the onboard hot memory layer, swaps its contents to the warm memory layer, copies the frequently accessed busy data page from the warm memory layer to the onboard hot memory layer, and updates the MMU page table to map the page's virtual address to a new physical address;

[0040] If it is determined that there is idle data in the GPU's onboard memory that has not been accessed for a long time, it will be migrated to the warm memory layer.

[0041] Thirdly, embodiments of this application provide a general-purpose graphics processor (GPGPU), including a GPU chip, GPU onboard memory, a driver module, an interconnect interface module, a memory, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the steps of the GPGPU memory expansion method based on the high-speed interconnect bus described above.

[0042] Fourthly, embodiments of this application provide an external memory pool, including a memory control chip, an interconnect interface module, an external memory pool, and a computer program that can run on the memory control module of the memory control chip. When the memory control module executes the program, it implements the steps of the GPGPU memory expansion method based on the high-speed interconnect bus described above.

[0043] Compared with existing technologies, it has the following outstanding advantages:

[0044] 1) This invention proposes a novel high-speed interconnect bus and its protocol that support memory expansion design. Unlike general-purpose buses such as PCIe or Ethernet, this bus's physical layer can utilize various media such as optical fiber, high-speed coaxial cable, or differential pairs on a specific substrate, and is optimized for point-to-point memory access to achieve ultra-low latency and high bandwidth. Its logical protocol layer abstracts away complex network routing and device discovery functions, focusing on lightweight memory read / write / atomic operations, achieving access efficiency close to that of a memory bus.

[0045] Technical benefits: Significantly reduces the round-trip time of GPU accessing external memory, from microseconds in PCIe to hundreds of nanoseconds or even lower. It also provides transfer capabilities on par with onboard memory bandwidth (e.g., TB / s), thereby minimizing GPU core waiting and idle time.

[0046] 2) The method and system of this invention propose an externally decoupled memory pool containing an independent intelligent controller. This memory pool is an independent hardware unit capable of accommodating a massive amount of standard memory (such as DDR5 / 6) or storage-class memory (SCM) far exceeding the capacity of a GPU board. Its core is an intelligent memory controller and a high-speed interconnect interface. This controller is responsible for communicating with the aforementioned high-speed interconnect bus via the high-speed interconnect interface, and for translating memory transaction requests transmitted on the bus into standard JEDEC memory commands, as well as managing address mapping, refresh, leveling, and error correction operations within the memory pool.

[0047] Technical benefits: It achieves physical separation and independent expansion (decoupling) of GPU computing resources and video memory resources. Users can flexibly configure terabyte-level massive "warm" video memory for the computing system as if adding storage devices, and the cost of the memory pool is far lower than that of HBM video memory of the same capacity, making it extremely cost-effective.

[0048] 3) This invention proposes a hardware-software co-operational, application-transparent memory management and address mapping mechanism. The invention includes a set of high-speed hardware interconnect interfaces and a modified driver software. The high-speed hardware interconnect interfaces are responsible for protocol conversion and physical connections. The driver software layer creates a layered memory management model within the operating system kernel, treating onboard HBM / GDDR video memory as hot data and external memory pools as warm data. It maps both to the GPU's virtual address space and can automatically and transparently migrate data pages between the two layers based on the frequency and pattern of data access.

[0049] Technical benefits: Completely transparent to upper-layer applications (such as PyTorch, TensorFlow, CUDA programs). Programmers can allocate and use memory space far exceeding the physical memory of a single graphics card without modifying existing code. The system's automatic data tiering and migration mechanism ensures that the vast majority of frequently accessed data hits the high-speed onboard memory, thus achieving average access performance close to that of onboard memory on a macro level, while enjoying the huge capacity of the external memory pool. Attached Figure Description

[0050] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0051] Figure 1 This is a schematic diagram of the GPGPU memory expansion system based on a high-speed interconnect bus according to the present invention;

[0052] Figure 2 This is a schematic diagram of a GPGPU memory expansion architecture based on a high-speed interconnect bus, according to an embodiment of the present invention.

[0053] Figure 3 This is a schematic diagram of the GPGPU memory expansion method based on a high-speed interconnect bus according to the present invention;

[0054] Figure 4 This is a schematic diagram of a GPGPU memory expansion method based on a high-speed interconnect bus according to an embodiment of the present invention. Detailed Implementation

[0055] In this invention, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of a single item or a plurality of items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be a single item or multiple items.

[0056] It should also be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0057] It should also be understood that, in various embodiments of the present invention, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0058] In the several embodiments provided by this invention, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0059] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0060] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0061] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0062] To make the above-mentioned features and effects of the present invention clearer and easier to understand, specific embodiments are described below in conjunction with the accompanying drawings. This specification discloses one or more embodiments incorporating the features of the present invention. The disclosed embodiments are merely illustrative. The scope of protection of the present invention is not limited to the disclosed embodiments, but is defined by the appended claims.

[0063] The following are system embodiments corresponding to the above method embodiments. This embodiment can be implemented in conjunction with the above embodiments. The relevant technical details mentioned in the above embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiments.

[0064] The inventors of this application faced long-term limitations due to the single-GPU memory capacity in their research project on large-scale AI model pre-training and deployment. When using a multi-GPU parallel solution, the team spent considerable effort adapting model partitioning and distributed training, but communication overhead remained an unavoidable bottleneck. Subsequently, the team attempted to use unified memory to overflow data into the system's main memory, but tests revealed that model training speed decreased by more than an order of magnitude, making project progress virtually impossible.

[0065] In analyzing the reasons for the failure, the inventors realized that the fundamental technical challenge lay in the lack of a dedicated data channel between the GPU and remote memory that combined "high bandwidth" and "ultra-low latency." PCIe was originally designed to connect various peripheral devices, and its protocol stack, designed for compatibility and versatility, is relatively complex, with long handshake and transaction processing flows, leading to unavoidable high latency. This is fatal for GPGPU computing, which requires frequent, fast, and random access to massive amounts of data.

[0066] This led the inventors to consider: could they bypass the limitations of PCIe and design a bus for the GPU similar to CPU-to-DDR memory? This idea faced significant technical challenges:

[0067] Protocol design challenges: A novel, minimalist, memory-semantic communication protocol needs to be designed. This protocol must eliminate unnecessary overhead from the general-purpose bus, enabling direct and fast transmission of addresses and data, while also incorporating reliable error checking and retransmission mechanisms.

[0068] System integration challenge: How can the GPU's memory controller recognize and manage this external memory pool without "modifying" the GPU core? This requires collaborative design at the hardware interface, firmware, and driver software levels.

[0069] The inventors conceived an innovative solution: designing a lightweight, point-to-point memory transaction protocol. Simultaneously, a set of "protocol bridging" interfaces was designed on both the GPU and memory pool sides. These interfaces reuse the GPU's internal high-speed bus on the GPU side while simultaneously implementing a new bus protocol for external communication with the external memory pool. Externally, an independent memory pool with an intelligent controller was designed. This controller manages the large-capacity DDR / SCM memory and interacts with the new bus protocol through a new high-speed interconnect interface. Finally, by modifying the GPU driver, this external memory pool is mapped into the GPU's address space, presenting upper-layer applications with a seamlessly expanded, single, massive video memory space. This series of concepts ultimately forms the core technical solution of this invention.

[0070] This invention discloses a GPGPU memory expansion system and method based on a high-speed interconnect bus. The system achieves transparent expansion of GPU memory by constructing an independent, large-capacity memory pool outside a single GPU and utilizing the GPU's high-speed interconnect bus. The system includes a GPU-side high-speed interconnect interface module, an external memory controller module, and an external memory pool. The interface module employs a high-speed interconnect protocol, providing bandwidth on par with onboard memory. The external controller translates bus transactions into standard DDR commands, enabling efficient management of massive general-purpose memory. The driver software constructs a unified memory address space and, through a hierarchical memory management mechanism, treats onboard memory as a "hot layer" and external memory as a "warm layer," supporting automatic page-level migration. This invention overcomes the limitation of single-GPU memory capacity, expanding memory to terabyte levels while maintaining high performance and significantly reducing system costs, making it valuable for applications in deep learning training, large-scale inference, and scientific computing.

[0071] This invention proposes a memory expansion scheme for general-purpose computing (GPGPU) scenarios. This scheme connects to an external large-capacity memory pool via a novel high-speed interconnect bus, aiming to overcome the limitations of single-GPU physical memory capacity. This invention has significant application value and broad market prospects in fields requiring massive memory support, such as artificial intelligence model training and inference, scientific computing, large-scale data visualization, and physics simulation.

[0072] The system of this application embodiment will be described in detail below with reference to specific embodiments:

[0073] like Figure 1 and Figure 2 As shown, this invention provides a detailed description of a GPGPU memory expansion method and system based on a high-speed interconnect bus. It should be clarified that the specific implementation methods described are only for illustrating the technical solution of this invention and are not intended to limit the invention.

[0074] This invention proposes a GPGPU memory expansion system based on a high-speed interconnect bus. The system includes: a GPU chip 10, a memory control chip 20 for an external memory pool 201; and

[0075] High-speed interconnect bus 30: used to connect GPU chip 10 and memory controller chip 20;

[0076] Interconnect interface module 40: Located on the GPU chip side and the memory controller chip side respectively, and connected through a high-speed interconnect bus. The interface module adopts a high-speed interconnect protocol to provide bandwidth of the same level as the onboard video memory.

[0077] Memory control module 202: Located inside the memory control chip 30, connected to the interconnect interface module 40 on the memory control chip side, and connected to the external memory pool 201. It is used to translate memory requests in bus transactions and manage the external memory pool 201.

[0078] Driver module 101: Located within GPU chip 10, it is used to create a memory management model, layer the GPU's onboard video memory and external memory pool, and uniformly map them into the GPU's virtual address space.

[0079] In this embodiment of the invention, the high-speed interconnect bus 30 adopts a high-speed interconnect protocol, which is divided into a physical layer and a logical protocol layer. The logical protocol layer is used to perform lightweight memory read / write / atomic operation transactions.

[0080] In this embodiment of the invention, the interconnection interface module 40 includes:

[0081] Protocol processing engine module: used to encapsulate memory access requests from the GPU into lightweight data packets, and unpack the data packets received from the bus and send them to the memory control module for use;

[0082] Physical layer control module: used to manage signal encoding and decoding, clock data recovery, and link establishment and maintenance.

[0083] In this embodiment of the invention, the interconnect interface module 40 includes: an interface 401 disposed on the GPU side, which reuses the high-speed bus inside the GPU and is connected to the GPU's cache and memory controller; and an interface 402 disposed on the external memory pool side, which is connected to the external memory pool side through a new bus protocol for communication with the outside.

[0084] In this embodiment of the invention, the memory control module 202 communicates with the high-speed interconnect bus through the interface 402 on the external memory pool side, translates the memory transaction requests transmitted on the bus into standard memory commands, and manages the address mapping, refresh, leveling and error correction operations within the memory pool.

[0085] In this embodiment of the invention, the driver module 101 creates a hierarchical memory management model, using the GPU's onboard video memory as hot data and the external memory pool as warm data. The hot data and warm data are mapped to the GPU's virtual address space, and page-level automatic migration can be performed between the two levels according to the frequency and pattern of data access.

[0086] Specifically, such as Figure 2 The diagram illustrates a system architecture of a preferred embodiment of the present invention. The system mainly includes: a GPGPU computing unit 1, a GPU-side high-speed interconnect interface 2, a high-speed bidirectional bus and a custom protocol 3, an external memory controller module 4, and an external memory pool 5.

[0087] GPGPU Computing Unit 1: This can be any commercial or custom graphics processor that supports GPGPU computing. Internally, it contains computing cores (SMs / CUs), L1 / L2 cache, and an onboard memory controller, and is connected to high-speed onboard memory (such as HBM3 or GDDR6X), which we call hot memory.

[0088] High-speed interconnect interface 2: This is one of the core components of this invention, encapsulated in the GPU and memory controller chip respectively (2.1 and 2.2). This module directly connects to the GPU's L2 cache and memory controller via a high-speed SerDes (serializer / deserializer) multiplexing of the GPU's high-speed I / O bus. Internally, this module includes:

[0089] Protocol processing engine: responsible for encapsulating memory access requests (address, read / write commands, data) from the GPU into lightweight data packets, wherein the lightweight data packets contain at least the request type, target address, data length and optional data fields, and parsing the corresponding data packets received from the bus and supplying them to the memory control module.

[0090] Physical layer controller (PHY): manages signal encoding and decoding (such as PAM4), clock data recovery (CDR), and link establishment and maintenance.

[0091] High-speed bidirectional bus and custom protocol 3: Located between the high-speed interconnect interface of the GPU and the memory controller chip, it adopts a point-to-point high-speed bidirectional bus and a corresponding simplified custom protocol. The protocol is based on a data packet format with a limited set of fields to realize the transmission of memory access requests and responses, and does not involve multi-master arbitration, cache consistency or general interconnect functions, thereby realizing low-latency data exchange.

[0092] External memory controller module 4: Together with the high-speed interconnect interface 2.2 on the memory controller chip side, it forms the memory controller chip, connecting externally to a large number of DDR memory modules in the memory pool. It also includes a signal conversion unit and a protocol processing engine. Its key function is to act as a "translator," converting memory requests in the bus protocol of this invention (e.g., reading 64 bytes of data at address 0x12345678) into timing and commands (such as row activation, column read / write, etc.) of the standard DDR memory protocol (JEDEC specification) used internally by the memory pool. This controller can also integrate a small local cache for temporarily storing frequently accessed data or prefetching data.

[0093] As described above, the system of the present invention can be implemented well.

[0094] Compared with the prior art, the present invention has the following outstanding advantages and beneficial effects:

[0095] 1) Huge breakthrough in capacity: Increased the available video memory capacity of GPGPU from tens of GB to TB, completely solving the demanding requirements of large AI models and scientific computing tasks on video memory capacity.

[0096] 2) Guarantee of high performance: Through high-speed interconnect bus and protocol, the access latency of extended video memory is significantly reduced and extremely high bandwidth is provided, avoiding the performance bottleneck of traditional PCIe solutions, enabling the GPU to make efficient use of the massive amount of extended video memory.

[0097] 3) Improved cost-effectiveness: Allowing the use of relatively inexpensive standard DDR memory as an expansion pool greatly reduces the hardware cost of obtaining massive amounts of video memory, making it extremely cost-effective compared to purchasing multiple top-tier GPUs.

[0098] 4) Architectural flexibility and decoupling: It achieves decoupling of computing and storage, allowing users to independently upgrade GPU computing power or expand video memory capacity, providing unprecedented flexibility for resource configuration in data centers and high-performance computing clusters.

[0099] 5) Ease of use and compatibility: Transparent to upper-layer applications, requiring no modification to the existing software ecosystem and programming model, greatly reducing the barrier to entry and facilitating rapid promotion and application.

[0100] Second aspect

[0101] like Figure 3 and Figure 4 As shown in the figure, this application provides a GPGPU memory expansion method based on a high-speed interconnect bus, the method including:

[0102] Initialization step 101: Initialize the interconnect interface module on the GPU side, and establish a link with the external memory controller chip through the high-speed interconnect bus. Initialize the interconnect interface module and memory controller module on the memory controller chip side.

[0103] Driver loading and memory space mapping step 102: The operating system loads the driver module. The driver module modifies the page table structure of the GPU's memory management unit at the kernel level, and maps the physical address space continuously to the high address segment of the GPU's virtual address space. The virtual address space is the video memory address space, which includes the onboard hot memory layer and the external warm memory layer.

[0104] Application memory allocation request step 103: When the driver module receives an external memory request and determines that the memory request space exceeds the GPU onboard video memory, the driver module allocates a virtual address and the corresponding physical page for the memory request from the address space mapped to the external memory layer.

[0105] Data access step 104: The GPU's computing module issues an operation request for a virtual address. The GPU performs the corresponding address translation, maps it to the external warm memory layer, and sends the operation request to the GPU-side interconnect interface module for encapsulation and then to the interconnect bus. The interconnect interface module on the memory controller chip side receives the operation request and passes it to the memory controller module for reading and processing. The external memory pool returns the data to the memory controller module, which encapsulates the data and sends it back to the GPU-side high-speed interconnect interface through the interconnect interface on the memory controller chip side and the interconnect bus. The GPU-side interconnect interface receives the response data packet, unpacks it, and submits the data to the GPU's computing module that issued the request.

[0106] In this embodiment of the invention, the above-mentioned GPGPU memory expansion method based on a high-speed interconnect bus further includes:

[0107] Intelligent data tiered migration step 105: The driver module monitors the access frequency of data pages located in the warm memory layer through the data access counter; if the access frequency of a certain data page exceeds the preset threshold, the driver module starts direct memory access operation, finds a free or cold data page in the onboard hot memory layer, swaps its contents to the warm memory layer, copies the frequently accessed busy data page from the warm memory layer to the onboard hot memory layer, and updates the MMU page table to map the page's virtual address to the new physical address;

[0108] If it is determined that there is idle data in the GPU's onboard memory that has not been accessed for a long time, it will be migrated to the warm memory layer.

[0109] Specifically, the working method flow of this embodiment is as follows: Figure 4 As shown, the specific steps are as follows:

[0110] Step S1: System Initialization. When the system starts, the GPU's BIOS or firmware first initializes the onboard interface 2.1, and establishes a link with the external memory controller chip through bus 3, initializing the 2.2 high-speed interconnect interface and memory controller module.

[0111] Step S2: Driver Loading and Memory Space Mapping. The operating system loads the specially designed GPU driver provided by this invention. The driver detects the existence and capacity of the external memory pool. Subsequently, the driver modifies the page table structure of the GPU's Memory Management Unit (MMU) at the kernel level, mapping the physical address space continuously to the high-order address segments of the GPU's virtual address space. At this point, from the perspective of the GPU and upper-layer applications, there exists a large, unified video memory address space, which includes the onboard hot memory layer and the external warm memory layer.

[0112] Step S3: Application memory allocation request. When an AI application (such as TensorFlow) requests the allocation of a large tensor, such as a weight matrix of size 500GB, the driver receives the request. Since this size exceeds the onboard video memory, the driver allocates a virtual address and a corresponding physical page for this tensor from the address space mapped to the warm memory layer.

[0113] Step S4: Data Access (taking a read operation as an example). When a CUDA Kernel is executed, the computing core on the GPU needs to read a portion of the data in the 500GB tensor.

[0114] S4a: Address Translation. The compute core issues a read request for a virtual address. The GPU's MMU performs address translation and finds that the physical page corresponding to the address is located in an external warm memory layer.

[0115] S4b: Request Encapsulation and Transmission. The read request is forwarded to interface 2.1 on the GPU side. The protocol processing engine encapsulates it into a read transaction data packet (containing information such as the target physical address and length) and sends it to bidirectional bus 3.

[0116] S4c: Request reception and processing. Data packets are transmitted via bidirectional bus 3 to the corresponding interconnect interface 2.2 of the memory controller chip and further passed to the memory controller module 4. The memory controller module receives and unpacks the packets, translating them into a series of read operation commands for the DDR5 memory chips.

[0117] S4d: Data Return. The DDR5 memory returns data to controller 4. The controller encapsulates this data into one or more response packets and sends them back to the GPU-side high-speed interconnect interface 2.1 via the high-speed interconnect interface 2.2 on the memory controller chip side and the bidirectional bus 3.

[0118] S4e: Data Reception and Submission. The high-speed interconnect interface 2.1 on the GPU side receives the response data packet, unpacks it, and submits the data to the GPU computing core that issued the request. The entire process is completed automatically at the hardware level and is transparent to the computing core.

[0119] Step S5 (Optional Optimization): Intelligent Data Layer Migration. The driver maintains a data access counter in the background, monitoring the access frequency of data pages located in the warm memory layer. When the driver detects that the access frequency of a data page (e.g., a layer in the model that is frequently used for computation) exceeds a preset threshold, it initiates a background DMA (Direct Memory Access) operation. This operation finds a free or "cold" data page in the onboard hot memory layer, swaps its contents to the warm memory layer, copies the frequently accessed "hot" data page from the warm memory layer to the onboard hot memory layer, and updates the MMU page table to map the page's virtual address to the new physical address. Conversely, "cold" data that has not been accessed for a long time in onboard memory is also migrated to the warm memory layer to free up valuable high-speed video memory space. Through this caching mechanism, it is ensured that the GPU core accesses high-speed video memory for the vast majority of computation time, thus achieving a balance between capacity and performance.

[0120] Thirdly, embodiments of this application provide a general-purpose graphics processor (GPGPU), including a GPU chip, GPU onboard memory, a driver module, an interconnect interface module, a memory, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the steps of the GPGPU memory expansion method based on the high-speed interconnect bus described above.

[0121] Fourthly, embodiments of this application provide an external memory pool, including a memory control chip, an interconnect interface module, an external memory pool, and a computer program that can run on the memory control module of the memory control chip. When the memory control module executes the program, it implements the steps of the GPGPU memory expansion method based on the high-speed interconnect bus described above.

[0122] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0123] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A GPGPU memory expansion system based on a high-speed interconnect bus, characterized in that, The system includes: a GPU chip, a memory control chip for an external memory pool; and High-speed interconnect bus: used to connect the GPU chip and the memory controller chip; Interconnect interface module: Located on the GPU chip side and the memory control chip side respectively, and connected through the high-speed interconnect bus. The interface module adopts a high-speed interconnect protocol to provide bandwidth of the same magnitude as the onboard video memory. Memory control module: Located within the memory control chip, connected to the interconnect interface module on the memory control chip side, and connected to the external memory pool externally, used to translate memory requests in bus transactions and manage the external memory pool; Driver module: Located within the GPU chip, it is used to create a memory management model, which separates the GPU's onboard video memory and external memory pool into layers and maps them uniformly into the GPU's virtual address space.

2. The GPGPU memory expansion system based on a high-speed interconnect bus according to claim 1, characterized in that, The high-speed interconnect bus adopts a high-speed interconnect protocol, which is divided into a physical layer and a logical protocol layer. The logical protocol layer is used to perform lightweight memory read / write / atomic operation transactions.

3. The GPGPU memory expansion system based on a high-speed interconnect bus according to claim 1, characterized in that, The interconnect interface module includes: Protocol processing engine module: used to encapsulate memory access requests from the GPU into lightweight data packets, and unpack data packets received from the bus and send them to the memory control module for use; Physical layer control module: used to manage signal encoding and decoding, clock data recovery, and link establishment and maintenance.

4. The GPGPU memory expansion system based on a high-speed interconnect bus according to claim 1, characterized in that, The interconnect interface module includes: interfaces respectively disposed on the GPU side, which reuse the high-speed bus inside the GPU on the GPU side, and are connected to the GPU's cache and memory controller; and The interface located on the external memory pool side is connected to the external memory pool side via a new bus protocol for communication with the outside world.

5. The GPGPU memory expansion system based on a high-speed interconnect bus according to claim 1, characterized in that, The memory control module communicates with the high-speed interconnect bus through the interface on the external memory pool side, translates the memory transaction requests transmitted on the bus into standard memory commands, and manages the address mapping, refresh, leveling and error correction operations within the memory pool.

6. The GPGPU memory expansion system based on a high-speed interconnect bus according to claim 1, characterized in that, The driver module creates a hierarchical memory management model, treating the GPU's onboard video memory as hot data and the external memory pool as warm data. It maps the hot data and warm data into the GPU's virtual address space and can automatically migrate pages between the two levels based on the frequency and pattern of data access.

7. A method for expanding GPGPU memory based on a high-speed interconnect bus, characterized in that, The method includes: Initialization steps: Initialize the interconnect interface module on the GPU side, and establish a link with the external memory control chip through the high-speed interconnect bus; initialize the interconnect interface module and the memory control module on the memory control chip side. Driver loading and memory space mapping steps: The operating system loads the driver module, and the driver module modifies the page table structure of the GPU's memory management unit at the kernel level to continuously map the physical address space to the high address segment of the GPU's virtual address space. The virtual address space is the video memory address space, which includes the onboard hot memory layer and the external warm memory layer. Application memory allocation request steps: When the driver module receives an external memory request and determines that the memory request space exceeds the GPU onboard video memory, the driver module allocates a virtual address and a corresponding physical page for the memory request from the address space mapped to the external memory layer; Data access steps: The GPU's computing module issues a virtual address operation request. The GPU performs corresponding address translation, mapping it to an external warm memory layer. The operation request is encapsulated by the GPU-side interconnect interface module and sent to the interconnect bus. The interconnect interface module on the memory controller chip side receives the operation request and passes it to the memory controller module for reading processing. The external memory pool returns data to the memory controller module, which encapsulates the data and sends it back to the GPU-side high-speed interconnect interface through the interconnect interface on the memory controller chip side and the interconnect bus. The GPU-side interconnect interface receives the response data packet, unpacks it, and submits the data to the GPU's computing module that issued the request.

8. The GPGPU memory expansion method based on a high-speed interconnect bus according to claim 7, characterized in that, The method further includes: Intelligent data tiered migration steps: The driver module monitors the access frequency of data pages located in the warm memory layer through a data access counter; if the access frequency of a certain data page exceeds a preset threshold, the driver module initiates a direct memory access operation, finds an idle or cold data page in the onboard hot memory layer, swaps its contents to the warm memory layer, copies the frequently accessed busy data page from the warm memory layer to the onboard hot memory layer, and updates the MMU page table to map the virtual address of the page to a new physical address; If it is determined that there is idle data in the GPU's onboard memory that has not been accessed for a long time, it will be migrated to the warm memory layer.

9. A general-purpose graphics processing unit (GPGPU), comprising a GPU chip, GPU onboard memory, a driver module, an interconnect interface module, a memory, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the GPGPU memory expansion method based on a high-speed interconnect bus as described in any one of claims 7-8.

10. An external memory pool, comprising a memory controller chip, an interconnect interface module, an external memory pool, and a computer program that can run on the memory controller module of the memory controller chip, characterized in that, When the memory control module executes the program, it implements the steps of the GPGPU memory expansion method based on a high-speed interconnect bus as described in any one of claims 7-8.

Images