Mcm-gpu cache management method and mcm-gpu

Abstract

The application discloses a kind of MCM-GPU last stage cache management method and MCM-GPU, the method includes: in GPU module, part LLC is selected as remote LLC, to cache the data stored in read-only storage space of remote GPU module;In response to the program executed, whether real-time judgment has stream processor initiates memory request;If yes, then judge whether memory request accesses read-only storage space, if not, then memory request is sent to the LLC in the corresponding LLC in current GPU module or the LLC in the corresponding remote GPU module, if yes, then judge whether memory request accesses local storage space;If yes, then memory request is sent to the LLC in the corresponding LLC in current GPU module;If not, then memory request is sent to the LLC in the corresponding remote LLC in current GPU module;Judge whether hit, if not, then memory request is sent to the LLC in the corresponding remote GPU module.The application can reduce the number of local access remote storage space and overhead, reduce on-chip network conflict, improve the performance of MCM-GPU.

Description

Technical Field

[0001] This invention relates to the field of GPU technology, and in particular to an MCM-GPU final-level cache management method and an MCM-GPU. Background Technology

[0002] With the continuous development of GPU (Graphics Processing Unit) technology, the number of GPU cores and computing power are constantly increasing. For example, from the Fermi to the Volta architecture, the number of Streaming Multiprocessors (SMs) in a GPU has increased from 14 to 80. Due to the limitations of manufacturing processes and chip size, integrating a large number of SMs on a single chip is becoming increasingly difficult. In order to further improve GPU performance and avoid the constraints of chip size, a multi-chip module GPU (MCM-GPU) has been proposed. MCM-GPU uses a multi-module packaging mode to package several GPU modules together to form a new chip. Depending on the packaging technology, multiple GPU modules communicate with each other through multi-layer interconnect substrate traces, interposer layer traces, and other technologies. Each GPU module is connected to DRAM (Dynamic Random Access Memory). The DRAM directly connected to the GPU module is the local storage space of that GPU module, while the DRAM directly connected to other GPU modules is the remote storage space of that GPU module.

[0003] refer to Figure 1 , Figure 1The diagram illustrates the structure of an MCM-GPU as an example. In an MCM-GPU, the last-level cache (LLC) of a GPU module typically employs two architectural approaches: a private last-level cache design and a shared last-level cache design. In the private last-level cache design, each GPU module's LLC is set as its private LLC. Memory access requests from stream processors within that GPU module first access the LLC of that GPU module. If a hit occurs, the data is returned; otherwise, the corresponding memory space is accessed. In the shared last-level cache design, stream processors from different GPU modules share all last-level caches within the MCM-GPU. Only one copy of the same data exists in the last-level cache, significantly improving the last-level cache hit rate. Specifically, when using the shared last-level cache design, if a stream processor in a GPU module initiates a memory access request, the memory access address is first used to determine which GPU module's memory space contains the requested data. Then, the memory access request is sent to the LLC of the corresponding GPU module. Typically, in a GPU module, one storage controller corresponds to multiple LLCs. The memory access request hashes the address of the data to be accessed to obtain the LLC ID. The corresponding LLC is selected for access based on the LLC ID. If the LLC is matched, the data is returned to the stream processor that initiated the request; otherwise, if the LLC fails, the memory access request is sent to the storage controller, the data is retrieved, stored in the LLC, and returned to the stream processor that initiated the request.

[0004] However, while using a shared last-level cache can save storage space, if a stream processor of a GPU module needs to access data in the last-level cache of another GPU module, these requests need to reach the last-level cache of the corresponding GPU module through the interconnection network between GPU modules. At this time, a large number of parallel memory access requests will generate intense on-chip network conflicts, resulting in a decrease in the performance of MCM-GPU. Summary of the Invention

[0005] To address some or all of the technical problems existing in the prior art, the present invention provides an MCM-GPU final-level cache management method and an MCM-GPU.

[0006] The technical solution of the present invention is as follows:

[0007] In a first aspect, a method for managing the final-level cache of an MCM-GPU is provided. The method is used with an MCM-GPU, which includes multiple interconnected GPU modules. The method includes:

[0008] A preset number of final-level caches are selected in each GPU module as remote final-level caches, and the data stored in the read-only storage space of the remote GPU module is cached using the remote final-level caches.

[0009] In response to the executing program, it determines in real time whether a stream processor has initiated a memory access request;

[0010] If a stream processor in a GPU module initiates a memory access request, determine whether the memory access request accesses read-only memory.

[0011] If it is not accessing read-only memory, the memory access request is sent to the corresponding last-level cache in the current GPU module or the corresponding last-level cache in the remote GPU module according to the memory access address. If it is accessing read-only memory, it is determined whether the memory access request accesses local memory.

[0012] If accessing local storage space, the memory access request is sent to the corresponding last-level cache in the current GPU module according to the memory access address;

[0013] If the access is not to local storage, the memory access request will be sent to the corresponding remote last-level cache in the current GPU module according to the memory access address.

[0014] Determine if the remote last-level cache has been hit. If not, send the memory access request to the corresponding last-level cache in the remote GPU module based on the memory access address.

[0015] In some possible implementations, the memory access request is sent to the corresponding last-level cache in the current GPU module or the last-level cache in the corresponding remote GPU module based on the memory access address, including:

[0016] Determine the storage space accessed by the memory access address. If it is local storage space, send the memory access request to the corresponding last-level cache in the current GPU module. If it is remote storage space, send the memory access request to the last-level cache in the corresponding remote GPU module.

[0017] In some possible implementations, the method further includes:

[0018] A routing control module is set up in the GPU module to determine whether the memory access request accesses read-only storage space and whether the memory access request accesses local storage space, and based on the determination result, the memory access request is sent to the last-level cache of the current GPU module, the remote last-level cache of the current GPU module, or the last-level cache in the remote GPU module.

[0019] In some possible implementations, the method further includes:

[0020] When writing GPU programs, if the allocated memory space is read-only storage space, a preset keyword is inserted at a preset location during program compilation to indicate that the memory access ld instruction is an instruction that accesses read-only storage space.

[0021] Secondly, an MCM-GPU is also provided, which utilizes the MCM-GPU final-level cache management method described above for final-level cache management.

[0022] The main advantages of the technical solution of this invention are as follows:

[0023] The MCM-GPU final-level cache management method and MCM-GPU of the present invention reduce the number of local accesses to remote storage space and the overhead by selecting a portion of the final-level cache in the GPU module as a remote final-level cache to cache the data stored in the final-level cache of the remote GPU module, thereby reducing on-chip network conflicts and improving the performance of MCM-GPU. At the same time, storing read-only data only in the remote final-level cache of the GPU module can reduce the overhead caused by maintaining data consistency. Attached Figure Description

[0024] The accompanying drawings, which are included to provide a further understanding of embodiments of the invention and constitute a part of this invention, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0025] Figure 1 A schematic diagram of an MCM-GPU structure is provided as an example;

[0026] Figure 2 This is a flowchart of an embodiment of the MCM-GPU final-level cache management method of the present invention;

[0027] Figure 3 This is a schematic diagram of the structure of an MCM-GPU according to an embodiment of the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0029] The technical solutions provided by the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0030] refer to Figure 2In a first aspect, an embodiment of the present invention provides an MCM-GPU final-level cache management method. This method is used with an MCM-GPU, which includes multiple interconnected GPU modules. The method includes the following steps:

[0031] A preset number of final-level caches are selected in each GPU module as remote final-level caches, and the data stored in the read-only storage space of the remote GPU module is cached using the remote final-level caches.

[0032] In response to the executing program, it determines in real time whether a stream processor has initiated a memory access request;

[0033] If a stream processor in a GPU module initiates a memory access request, determine whether the memory access request accesses read-only memory.

[0034] If it is not accessing read-only memory, the memory access request is sent to the corresponding last-level cache in the current GPU module or the corresponding last-level cache in the remote GPU module according to the memory access address. If it is accessing read-only memory, it is determined whether the memory access request accesses local memory.

[0035] If accessing local storage space, the memory access request is sent to the corresponding last-level cache in the current GPU module according to the memory access address;

[0036] If the access is not to local storage, the memory access request will be sent to the corresponding remote last-level cache in the current GPU module according to the memory access address.

[0037] Determine if the remote last-level cache has been hit. If not, send the memory access request to the corresponding last-level cache in the remote GPU module based on the memory access address.

[0038] Specifically, in one embodiment of the present invention, in response to the program executed in the MCM-GPU, during the execution of the program, it is determined in real time whether a stream processor initiates a memory access request. If a stream processor in a GPU module initiates a memory access request, it is determined whether the memory access request accesses read-only memory. If it does not access read-only memory, the memory access request is sent to the corresponding last-level cache in the current GPU module or the last-level cache in the corresponding remote GPU module according to the memory access address. If the last-level cache is hit, data is returned to the stream processor that initiated the memory access request. If the last-level cache fails, the memory access request is sent to the storage controller connected to the corresponding GPU module, and the data is returned to the stream processor that initiated the memory access request. If read-only memory is accessed, it is further determined whether the memory access request accesses local storage space. If local storage space is accessed, the memory access request is sent to the current GPU module according to the memory access address. If the last-level cache in the GPU module is hit, data is returned to the stream processor that initiated the memory access request. If the last-level cache is missed, a memory access request is sent to the storage controller connected to the current GPU module, and the data is returned to the stream processor that initiated the memory access request. If it is not a local storage space access, the memory access request is sent to the corresponding remote last-level cache in the current GPU module according to the memory access address. If the remote last-level cache is hit, data is returned to the stream processor that initiated the memory access request. If not, the memory access request is sent to the last-level cache in the corresponding remote GPU module according to the memory access address. If the last-level cache is hit, data is returned to the stream processor that initiated the memory access request. If the last-level cache is missed, a memory access request is sent to the storage controller connected to the remote GPU module, and the data is returned to the stream processor that initiated the memory access request.

[0039] An embodiment of the present invention provides an MCM-GPU final-level cache management method that selects a portion of the final-level cache in the GPU module as a remote final-level cache to cache the data stored in the final-level cache of the remote GPU module. This reduces the number of times and overhead of local access to remote storage space, reduces on-chip network conflicts, and improves the performance of the MCM-GPU. At the same time, storing read-only data only in the remote final-level cache of the GPU module can reduce the overhead caused by maintaining data consistency.

[0040] Furthermore, in one embodiment of the present invention, the number of far-end caches in the GPU module used as far-end caches is set according to actual conditions. Generally, the more data the far-end cache needs to cache, the higher the required storage capacity of the far-end cache, and correspondingly, the more far-end caches are used as far-end caches.

[0041] Furthermore, in one embodiment of the present invention, sending the memory access request to the corresponding last-level cache in the current GPU module or the corresponding last-level cache in a remote GPU module according to the memory access address specifically includes:

[0042] Determine the storage space accessed by the memory access address. If it is local storage space, send the memory access request to the corresponding last-level cache in the current GPU module. If it is remote storage space, send the memory access request to the last-level cache in the corresponding remote GPU module.

[0043] Specifically, in one embodiment of the present invention, when it is necessary to send a memory access request to the last-level high-level cache, the ID of the last-level cache to be accessed is calculated based on the memory access address of the memory access request, and the memory access request is sent to the corresponding last-level cache based on the last-level cache ID.

[0044] Furthermore, in one embodiment of the present invention, the method further includes:

[0045] A routing control module is set up in the GPU module to determine whether the memory access request accesses read-only storage space and whether the memory access request accesses local storage space, and based on the determination result, the memory access request is sent to the last-level cache of the current GPU module, the remote last-level cache of the current GPU module, or the last-level cache in the remote GPU module.

[0046] By setting up a routing control module, it is possible to easily control the sending of memory access requests, thereby improving the management efficiency of the last-level cache.

[0047] The routing control module can be set in the on-chip interconnect network (NoC).

[0048] Caching data stored in a remote GPU in the local GPU module's remote final-level cache can lead to multiple copies of the same data, causing data consistency issues. In one embodiment of this invention, by caching only read-only data in the remote final-level cache, the overhead of maintaining data consistency can be effectively reduced. Furthermore, to facilitate the routing control module's determination of whether a memory access request accesses read-only storage space, thus improving management efficiency, in one embodiment of this invention, the method further includes: when writing the GPU program, if the allocated memory space is read-only storage space, inserting a preset keyword at a preset location during program compilation to indicate that the memory access ld instruction is an instruction accessing read-only storage space.

[0049] Specifically, during GPU program development, if the allocated memory space is read-only storage, the GPU program compiler inserts the `ld.readonly` keyword during compilation. The `ld.readonly` keyword indicates that the memory access instruction (`ld`) is for accessing read-only storage. When a memory access request is generated by `ld.readonly`, the routing control module determines that it is accessing read-only storage.

[0050] Secondly, an embodiment of the present invention also provides an MCM-GPU, which utilizes the above-described MCM-GPU final-level cache management method for final-level cache management.

[0051] refer to Figure 3 In addition to a GPU module, SM, NoC, LLC, and storage controller, the MCM-GPU provided in one embodiment of the present invention also includes a routing control module and a remote LLC. The routing control module is used to determine whether the memory access request accesses read-only storage space and whether the memory access request accesses local storage space, and based on the determination result, sends the memory access request to the LLC of the current GPU module, the remote LLC of the current GPU module, or the LLC in the remote GPU module. The remote LLC is used to cache the data stored in the read-only storage space of the remote GPU module.

[0052] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Additionally, the terms "front," "back," "left," "right," "upper," and "lower" in this document refer to the placement shown in the accompanying drawings.

[0053] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for managing the final-level cache of an MCM-GPU, characterized in that, The method is used for MCM-GPU, which includes multiple interconnected GPU modules, and the method includes: A preset number of final-level caches are selected in each GPU module as remote final-level caches, and the data stored in the read-only storage space of the remote GPU module is cached using the remote final-level caches. In response to the executing program, it determines in real time whether a stream processor has initiated a memory access request; If a stream processor in a GPU module initiates a memory access request, determine whether the memory access request accesses read-only memory. If it is not accessing read-only memory, the memory access request is sent to the corresponding last-level cache in the current GPU module or the corresponding last-level cache in the remote GPU module according to the memory access address. If it is accessing read-only memory, it is determined whether the memory access request accesses local memory. If accessing local storage space, the memory access request is sent to the corresponding last-level cache in the current GPU module according to the memory access address; If the access is not to local storage, the memory access request will be sent to the corresponding remote last-level cache in the current GPU module according to the memory access address. Determine if the remote last-level cache has been hit. If not, send the memory access request to the corresponding last-level cache in the remote GPU module based on the memory access address.

2. The MCM-GPU final-level cache management method according to claim 1, characterized in that, Based on the memory access address, the memory access request is sent to the corresponding last-level cache in the current GPU module or the corresponding last-level cache in a remote GPU module, including: Determine the storage space accessed by the memory access address. If it is local storage space, send the memory access request to the corresponding last-level cache in the current GPU module. If it is remote storage space, send the memory access request to the last-level cache in the corresponding remote GPU module.

3. The MCM-GPU final-level cache management method according to claim 1 or 2, characterized in that, The method further includes: A routing control module is set up in the GPU module to determine whether the memory access request accesses read-only storage space and whether the memory access request accesses local storage space, and based on the determination result, the memory access request is sent to the last-level cache of the current GPU module, the remote last-level cache of the current GPU module, or the last-level cache in the remote GPU module.

4. The MCM-GPU final-level cache management method according to claim 3, characterized in that, The method further includes: When writing GPU programs, if the allocated memory space is read-only storage space, a preset keyword is inserted at a preset location during program compilation to indicate that the memory access ld instruction is an instruction that accesses read-only storage space.

5. An MCM-GPU, characterized in that, The MCM-GPU uses the MCM-GPU final-level cache management method as described in any one of claims 1-4 to perform final-level cache management.

Images