Methods and systems for enabling GPU module in-memory computing through hybrid storage

Abstract

This invention provides a method and system for achieving GPU module in-memory computing through hybrid storage. The GPU module includes a GPU chip, which contains a DDR controller, a PCM controller, and SRAM. A CXL controller is located at the front end of the GPU chip. The data bus inside the GPU chip connects to the CXL controller, DDR controller, PCM controller, and SRAM. The CXL controller reports memory addresses to the CPU for access via the CXL protocol. The space of the reported memory addresses is mapped to the PCM address range managed by the GPU chip, and the space of the reported memory addresses is smaller than the PCM address space managed by the GPU chip. The CPU directly accesses the reported memory addresses and copies data from the system DDR to the reported memory addresses. This improves the efficiency of GPU data access, reduces GPU data access latency, and enhances the overall computing performance of the system.

Description

Technical Field

[0001] This invention relates to the field of solid-state drive technology, and in particular to a method and system for achieving GPU module in-situ computing through hybrid storage. Background Technology

[0002] PCM (Phase Change Memory) and NAND flash memory are two distinct storage technologies. While NAND dominates in terms of cost and large-capacity storage, PCM offers significant advantages in key performance metrics, making it particularly suitable for applications requiring high performance, high durability, and low latency. Its main advantages include: extremely fast read / write speeds (especially writes): tens to hundreds of times faster than NAND; ultra-high durability: write endurance far exceeds NAND, tens to hundreds of times longer; extremely low latency (especially writes): near-memory-level latency, with no block erase; byte-level addressing: achieves true random access, eliminating write amplification.

[0003] Based on the advantages of PCM media over NAND media, more and more high-performance storage scenarios or devices are beginning to use PCM media for data storage.

[0004] Reference Figure 1 In traditional GPU-based computing architectures, when the GPU needs data, it requires CPU scheduling to load the data from the SSD into system memory (DDR) and then copy it to the GPU module's video memory (DDR). This process involves multiple data copies and interactions between system components, which introduces latency. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a method and system for GPU module storage and computing through hybrid storage, thereby improving the efficiency of GPU data access, reducing the latency of GPU data access, and improving the overall computing performance of the system.

[0006] In a first aspect, embodiments of the present invention provide a method for achieving in-memory computing in a GPU module through hybrid storage, applied to a GPU module, the GPU module including a GPU chip, the GPU chip having a DDR controller, a PCM controller and SRAM disposed therein, a CXL controller disposed at the front end of the GPU chip, and a data bus inside the GPU chip respectively connecting the CXL controller, the DDR controller, the PCM controller and the SRAM; the method includes: The CXL controller reports the memory address to the CPU for access via the CXL protocol; wherein the space of the reported memory address is mapped to the PCM address range managed internally by the GPU chip, and the space of the reported memory address is smaller than the PCM address space managed internally by the GPU chip. The CPU directly accesses the reported memory address and copies the data from the system DDR to the reported memory address.

[0007] Furthermore, the storage addresses of all the data buses are uniformly addressed by the GPU chip, and the unified addressing of the GPU chip includes: the address range managed by the DDR controller, the address range managed by the PCM controller, and the address range of the SRAM.

[0008] Furthermore, the GPU chip interfaces with both the DDR chip and the PCM chip, using the DDR chip as video memory and the PCM chip as local module storage; the method further includes: The CPU selects the data accessed by the GPU module according to the upper-layer software algorithm and stores the accessed data in the PCM chip of the GPU module.

[0009] Furthermore, the method also includes: The GPU chip identifies the address range stored in the PCM chip based on the address mapping of the PCM chip; Retrieve the data corresponding to the current storage address from the address range stored in the PCM chip.

[0010] Furthermore, the method also includes: When the GPU module loses power, the GPU module flushes the cached data written to the PCM medium from the SRAM inside the GPU chip to the PCM medium so that the PCM medium can store the data.

[0011] Secondly, embodiments of the present invention provide a system for GPU module in-memory computing through hybrid storage. The system includes a GPU module and a CPU. The GPU module includes a GPU chip. The GPU chip is equipped with a DDR controller, a PCM controller and an SRAM. A CXL controller is provided at the front end of the GPU chip. The data bus inside the GPU chip is connected to the CXL controller, the DDR controller, the PCM controller and the SRAM respectively. The CXL controller is used to provide the CPU with access to the reported memory address via the CXL protocol; wherein the space of the reported memory address is mapped to the PCM address range managed internally by the GPU chip, and the space of the reported memory address is smaller than the PCM address space managed internally by the GPU chip. The CPU is used to directly access the reported memory address and copy data from the system DDR to the reported memory address.

[0012] Furthermore, the storage addresses of all the data buses are uniformly addressed by the GPU chip, and the unified addressing of the GPU chip includes: the address range managed by the DDR controller, the address range managed by the PCM controller, and the address range of the SRAM.

[0013] Furthermore, the GPU chip interfaces with both the DDR chip and the PCM chip, using the DDR chip as video memory and the PCM chip as local module storage; the system also includes: The CPU is used to select the data accessed by the GPU module according to the upper-layer software algorithm, and store the accessed data into the PCM chip of the GPU module.

[0014] Thirdly, embodiments of the present invention provide an electronic device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the computer program to implement the method described above.

[0015] Fourthly, embodiments of the present invention provide a computer-readable medium having processor-executable non-volatile program code that causes the processor to perform the method described above.

[0016] This invention provides a method and system for achieving in-memory computing in a GPU module through hybrid storage. The GPU module includes a GPU chip, which houses a DDR controller, a PCM controller, and SRAM. A CXL controller is located at the front end of the GPU chip. The data bus within the GPU chip connects to the CXL controller, DDR controller, PCM controller, and SRAM. The method includes: the CXL controller reporting memory addresses to the CPU via the CXL protocol; wherein the space of the reported memory addresses is mapped to the PCM address range managed internally by the GPU chip, and the space of the reported memory addresses is smaller than the PCM address space managed internally by the GPU chip; the CPU directly accesses the reported memory addresses and copies data from the system DDR to the reported memory addresses; this improves the efficiency of GPU data access, reduces GPU data access latency, and enhances the overall computing performance of the system.

[0017] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention are realized and obtained in accordance with the structures particularly pointed out in the description, claims and drawings.

[0018] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0019] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of a traditional GPU module; Figure 2 This is a flowchart of a method for implementing GPU module in-memory computing through hybrid storage, provided in Embodiment 1 of the present invention. Figure 3 This is a schematic diagram of a GPU module provided in Embodiment 1 of the present invention; Figure 4 This is a schematic diagram of another GPU module provided in Embodiment 1 of the present invention; Figure 5 This is a schematic diagram of the memory address reported by the CPU according to Embodiment 1 of the present invention; Figure 6 This is a schematic diagram of a system for GPU module in-memory computing through hybrid storage, provided in Embodiment 2 of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] This application adds a PCM chip to the GPU module as local storage, so that the GPU module can bypass the CPU and system DDR and directly send data requests to the module's local PCM storage. The data is directly transferred to the GPU module's video memory, which can improve the processing efficiency of the GPU module, expand the capacity of the GPU module's local storage data, and remove the data access bottleneck for high-bandwidth, low-latency applications such as AI.

[0023] The following explains the terminology used in this application: GPU module: The entire module board that includes the GPU chip and peripheral devices, usually including DDR video memory; GPU chip: The core chip on the module; DDR memory: refers to the DDR chips on the GPU module; System memory (DDR): refers to the DDR chips on the system motherboard that are directly connected to the CPU; DDR controller: refers to the DDR control circuit inside the chip, used to control the external DDR chip; PCM controller: refers to the PCM control circuit inside the chip, used to control the PCM chip outside the chip.

[0024] To facilitate understanding of this embodiment, the embodiments of the present invention will be described in detail below.

[0025] Example 1: Figure 2 This is a flowchart of a method for implementing GPU module in-memory computing through hybrid storage, as provided in Embodiment 1 of the present invention.

[0026] Reference Figure 2 The method includes the following steps: In step S101, the CXL controller reports the memory address to the CPU for access via the CXL protocol; wherein, the space of the reported memory address is mapped to the PCM address range managed internally by the GPU chip, and the space of the reported memory address is smaller than the PCM address space managed internally by the GPU chip. In step S102, the CPU directly accesses the reported memory address and copies the data from the system DDR to the reported memory address.

[0027] Specifically, the CXL controller inside the GPU chip can report a memory address range to the CPU via the CXL.mem protocol. The space of memory addresses reported via the CXL.mem protocol is mapped to the PCM address range managed internally by the GPU chip, and the address space reported via the CXL.mem protocol needs to be smaller than the PCM address space managed internally by the GPU chip (e.g., ...). Figure 5 The PCM provides a total address range of 256G, but only 250G of the address range is reported to the CPU via the CXL.mem protocol. The remaining 6G of the address range is used as storage for the GPU module's own system data and is not accessed externally. The CPU can directly access the memory address space reported by CXL.mem (as shown in Figure 5, which is the external 250G address range).

[0028] When the CPU copies data from the system DDR to the address range reported by CXL.mem, it is equivalent to the data being directly stored on the PCM medium of the GPU module.

[0029] Reference Figure 3By adding a PCM chip as local storage to the GPU module, the GPU module can bypass the CPU and system DDR and directly send data requests to the local PCM chip of the GPU module. The data is then directly transferred to the GPU module's video memory (DDR), which can improve the processing efficiency of the GPU and realize a new type of GPU module.

[0030] Reference Figure 4 It is used in GPU modules, which include GPU chips. The GPU chips contain DDR controllers, PCM controllers and SRAM. The front end of the GPU chips contains CXL controllers. The data buses inside the GPU chips are connected to the CXL controller, DDR controller, PCM controller and SRAM respectively.

[0031] Specifically, the GPU chip is internally designed to have both a DDR controller and a PCM controller (a controller similar to DDR but with some differences), and the GPU chip's front-end interface uses a CXL controller for CXL protocol communication.

[0032] All data bus memory addresses are uniformly addressed by the GPU chip. The GPU chip's uniform addressing includes the address range managed by the DDR controller, the address range managed by the PCM controller, and the address range of SRAM. All memory addresses are uniformly managed by the GPU chip.

[0033] Furthermore, the GPU chip interfaces with both the DDR chip and the PCM chip, using the DDR chip as video memory and the PCM chip as local module storage; the method also includes the following steps: In step S201, the CPU selects the data accessed by the GPU module according to the upper-layer software algorithm and stores the accessed data in the PCM chip of the GPU module.

[0034] Here, the CPU can select data that the GPU module frequently needs to access and store it in the module's PCM storage based on the upper-level software algorithm, instead of storing it in the system SSD.

[0035] Furthermore, the method also includes the following steps: Step S301: The GPU chip identifies the address range stored in the PCM chip based on the address mapping of the PCM chip. Step S302: Obtain the data corresponding to the current storage address from the address range stored in the PCM chip.

[0036] Specifically, the GPU chip and its firmware program are based on the address mapping of the PCM chip (e.g., Figure 5The GPU's internal addressing range (2) shown can identify which address ranges of data are stored on the GPU module's local PCM chip. When the GPU needs data in these address ranges, it can directly access the local PCM chip to move the required data to the DDR memory for processing, without having to move data from the remote SSD, thus improving data retrieval efficiency. For example, the GPU chip can identify the local DDR address range as 1 to 256G, and simultaneously identify the local PCM chip address range as 256G+1 to 512G. When data from 400G to 401G is needed, it can directly retrieve the data from the PCM address range.

[0037] Furthermore, the method also includes the following steps: Step S401: When the GPU module loses power, the GPU module flushes the cached data written to the PCM medium from the SRAM inside the GPU chip to the PCM medium so that the PCM medium can store the data.

[0038] Specifically, a small BBU (battery) is added to the GPU module to provide backup power for the entire module. When the GPU module loses power unexpectedly, the GPU can flush the cached data in the chip's internal SRAM that is ready to be written to the PCM medium to the PCM medium for storage. At the same time, by taking advantage of the non-volatility of the PCM medium, the data will not be lost even if the GPU module loses power unexpectedly, thus ensuring the integrity of the locally stored data.

[0039] In this application, when the GPU module accesses certain stored data, it can bypass the process of loading data from the SSD into system memory (DDR) and then copying it to the GPU module's video memory (DDR) through CPU scheduling. This allows the GPU to bypass the CPU and system DRAM and directly initiate data requests to the module's local PCM storage, thereby improving the efficiency of GPU data access, reducing GPU data access latency, improving the overall computing performance of the system, improving the overall AI (artificial intelligence) related performance of the system, and accelerating the widespread application of PCM, a high-performance and high-durability new medium.

[0040] Example 2: Figure 6 This is a schematic diagram of a system for GPU module in-memory computing through hybrid storage, provided in Embodiment 2 of the present invention.

[0041] Reference Figure 6 The system includes a GPU module and a CPU. The GPU module includes a GPU chip, which contains a DDR controller, a PCM controller and an SRAM. A CXL controller is located at the front end of the GPU chip. The data bus inside the GPU chip is connected to the CXL controller, the DDR controller, the PCM controller and the SRAM respectively. The CXL controller is used to report memory addresses to the CPU via the CXL protocol. The space of the reported memory addresses is mapped to the PCM address range managed by the GPU chip, and the space of the reported memory addresses is smaller than the PCM address space managed by the GPU chip. The CPU is used to directly access the reported memory address and copy data from the system DDR to the reported memory address.

[0042] Furthermore, the storage addresses of all data buses are uniformly addressed by the GPU chip. The unified addressing of the GPU chip includes the address range managed by the DDR controller, the address range managed by the PCM controller, and the address range of SRAM.

[0043] Furthermore, the GPU chip interfaces with both DDR and PCM chips, using the DDR chip as video memory and the PCM chip as local module storage; the system also includes: The CPU is used to select the data accessed by the GPU module according to the upper-level software algorithm and store the accessed data into the PCM chip of the GPU module.

[0044] This invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the method for achieving GPU module in-memory computing through hybrid storage provided in the above embodiments.

[0045] This invention also provides a computer-readable medium having processor-executable non-volatile program code, on which a computer program is stored. When the computer program is run by a processor, it executes the steps of the method for implementing GPU module in-memory computing through hybrid storage as described above.

[0046] The computer program product provided in this embodiment of the invention includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the preceding method embodiments. For specific implementation details, please refer to the method embodiments, which will not be repeated here.

[0047] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system and apparatus described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0048] Furthermore, in the description of the embodiments of the present invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in the present invention based on the specific circumstances.

[0049] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0050] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0051] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for implementing in-memory computing in a GPU module through hybrid storage, characterized in that, The method is applied to a GPU module, the GPU module including a GPU chip, the GPU chip having a DDR controller, a PCM controller and SRAM internally, a CXL controller on the front end of the GPU chip, and an internal data bus connecting the CXL controller, the DDR controller, the PCM controller and the SRAM respectively; the GPU chip simultaneously interfaces with the DDR chip and the PCM chip, using the DDR chip as video memory and the PCM chip as local storage of the module; the method includes: The CXL controller reports the memory address to the CPU for access via the CXL protocol; wherein the space of the reported memory address is mapped to the PCM address range managed internally by the GPU chip, and the space of the reported memory address is smaller than the PCM address space managed internally by the GPU chip. The CPU directly accesses the reported memory address and copies the data from the system DDR to the reported memory address.

2. The method for achieving GPU module in-memory computing through hybrid storage according to claim 1, characterized in that, The storage addresses of all the data buses are uniformly addressed by the GPU chip. The unified addressing of the GPU chip includes the address range managed by the DDR controller, the address range managed by the PCM controller, and the address range of the SRAM.

3. The method for achieving GPU module in-memory computing through hybrid storage according to claim 1, characterized in that, The method further includes: The CPU selects the data accessed by the GPU module according to the upper-layer software algorithm and stores the accessed data in the PCM chip of the GPU module.

4. The method for realizing GPU module in-memory computing through hybrid storage according to claim 1, characterized in that, The method further includes: The GPU chip identifies the address range stored in the PCM chip based on the address mapping of the PCM chip; Retrieve the data corresponding to the current storage address from the address range stored in the PCM chip.

5. The method for achieving GPU module in-memory computing through hybrid storage according to claim 1, characterized in that, The method further includes: When the GPU module loses power, the GPU module flushes the cached data written to the PCM medium from the SRAM inside the GPU chip to the PCM medium so that the PCM medium can store the data.

6. A system for enabling GPU module in-memory computing through hybrid storage, characterized in that, The system includes a GPU module and a CPU. The GPU module includes a GPU chip, which contains a DDR controller, a PCM controller, and SRAM. A CXL controller is located at the front end of the GPU chip. The data bus inside the GPU chip is connected to the CXL controller, the DDR controller, the PCM controller, and the SRAM, respectively. The GPU chip interfaces with both the DDR chip and the PCM chip, using the DDR chip as video memory and the PCM chip as local storage for the module. The CXL controller is used to provide the CPU with access to the reported memory address via the CXL protocol; wherein the space of the reported memory address is mapped to the PCM address range managed internally by the GPU chip, and the space of the reported memory address is smaller than the PCM address space managed internally by the GPU chip. The CPU is used to directly access the reported memory address and copy data from the system DDR to the reported memory address.

7. The system for GPU module in-memory computing via hybrid storage according to claim 6, characterized in that, The storage addresses of all the data buses are uniformly addressed by the GPU chip. The unified addressing of the GPU chip includes the address range managed by the DDR controller, the address range managed by the PCM controller, and the address range of the SRAM.

8. The system for GPU module in-memory computing via hybrid storage according to claim 6, characterized in that, The system also includes: The CPU is used to select the data accessed by the GPU module according to the upper-layer software algorithm, and store the accessed data into the PCM chip of the GPU module.

9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, characterized in that, When the processor executes the computer program, it implements the method described in any one of claims 1 to 5.

10. A computer-readable medium having processor-executable non-volatile program code, characterized in that, The program code causes the processor to execute the method according to any one of claims 1 to 5.

Images