MCM-gpu adaptive last level cache structure and cache switching method thereof

Abstract

The application discloses an MCM-GPU adaptive last-level cache structure arranged in a GPU module, comprising a Tag Array and a Date Array, wherein the Date Array is used for storing data, and the Tag Array is used for checking whether the data corresponding to an address is in the cache; the application further comprises a local memory access queue used for storing a memory access request of the current GPU module, a remote memory access queue used for storing a memory access request of other GPU modules, an LLC architecture change flag register used for storing an LLC architecture change flag indicating whether the architecture organization mode of the current last-level cache needs to be changed, and an LLC architecture flag register used for storing an LLC architecture flag indicating whether the current last-level cache is switched to a private last-level cache design or a shared last-level cache design. The application can support the dynamic switching of the shared last-level cache and the private last-level cache, can adaptively select the last-level cache architecture organization mode according to the configuration during program running, can meet the program memory access requirement, and can improve the performance of the MCM-GPU.

Description

Technical Field

[0001] This invention relates to the field of GPU technology, and in particular to an MCM-GPU adaptive final-level cache structure and its cache switching method. Background Technology

[0002] With the continuous development of GPU (Graphics Processing Unit) technology, the number of GPU cores and computing power are constantly increasing. For example, from the Fermi to the Volta architecture, the number of Streaming Multiprocessors (SMs) in a GPU has increased from 14 to 80. Due to the limitations of manufacturing processes and chip size, integrating a large number of SMs on a single chip is becoming increasingly difficult. In order to further improve GPU performance and avoid the constraints of chip size, a multi-chip module GPU (MCM-GPU) has been proposed. MCM-GPU uses a multi-module packaging mode to package several GPU modules together to form a new chip. Depending on the packaging technology, multiple GPU modules communicate with each other through multi-layer interconnect substrate traces, interposer layer traces, and other technologies. Each GPU module is connected to DRAM (Dynamic Random Access Memory). The DRAM directly connected to the GPU module is the local storage space of that GPU module, while the DRAM directly connected to other GPU modules is the remote storage space of that GPU module.

[0003] refer to Figure 1 , Figure 1 The diagram illustrates an example of an MCM-GPU architecture. In an MCM-GPU, the last-level cache (LLC) of a single GPU module typically employs two architectural approaches: a shared last-level cache design and a private last-level cache design. In the shared last-level cache design, stream processors from different GPU modules use the same last-level cache, ensuring that only one copy of the same data exists within it, significantly improving the last-level cache hit rate. In the private last-level cache design, each GPU module's LLC is set as its private LLC. Memory access requests from stream processors within that GPU module first access the local LLC. If a hit occurs, the data is returned; otherwise, the corresponding memory space is accessed. Compared to the shared last-level cache architecture, the private last-level cache architecture reduces communication between different GPU modules.

[0004] However, while using a shared last-level cache can save storage space, if a stream processor of one GPU module needs to access data in the last-level cache of another GPU module, these requests must reach the corresponding GPU module's last-level cache through the inter-GPU module interconnection network. This results in a large number of concurrent memory access requests causing severe on-chip network conflicts, leading to a degraded MCM-GPU performance. Using a private last-level cache requires storing data copies locally to reduce communication between different GPU modules. However, if the program has a large shared dataset, the numerous data copies will waste last-level cache space, leading to extremely high cache miss rates. A large number of cache miss requests need to access the corresponding GPU module's storage space through the inter-GPU module interconnection network, causing conflicts on the interconnection network and increasing memory access pressure, further degrading MCM-GPU performance. Summary of the Invention

[0005] To address some or all of the technical problems existing in the prior art, this invention provides an MCM-GPU adaptive final-level cache structure and its cache switching method.

[0006] The technical solution of the present invention is as follows:

[0007] Firstly, an MCM-GPU adaptive final-level cache structure is provided, set in the GPU module of the MCM-GPU, including: a Tag Array and a Date Array. The Date Array is used to store data, and the Tag Array is used to check whether the data corresponding to the address is in the cache. It also includes:

[0008] A local memory access queue, which is used to store the memory access requests of the current GPU module;

[0009] A remote memory access queue, which is used to store memory access requests from other GPU modules in the MCM-GPU;

[0010] LLC architecture change flag register is used to store LLC architecture change flags that indicate whether the current architecture organization of the last-level cache needs to be changed.

[0011] The LLC architecture flag register stores LLC architecture flags that indicate whether the current last-level cache should be switched to a private last-level cache design or a shared last-level cache design when the architecture organization of the last-level cache needs to be changed.

[0012] In some possible implementations, when the last-level cache uses a private last-level cache design, memory access requests issued by the stream processors in the current GPU module are placed in the local memory access queue, waiting to access the last-level cache.

[0013] In some possible implementations, when the last-level cache uses a shared last-level cache design, the memory access request is placed in the local memory access queue of the current GPU module or the remote memory access queue of other GPU modules according to the memory access address of the memory access request issued by the stream processor in the current GPU module, waiting to access the last-level high-level cache.

[0014] If the memory access address of the memory access request issued by the stream processor in the current GPU module accesses the local memory space, the memory access request is placed in the local memory access queue of the current GPU module.

[0015] If the memory access address of the memory access request issued by the stream processor in the current GPU module is a remote memory space, the memory access request is placed in the remote memory access queue of the GPU module corresponding to the remote memory space.

[0016] In some possible implementations, when the LLC architecture changes the flag to 0, it indicates that the current architecture organization of the last-level cache does not need to be changed.

[0017] When the LLC architecture changes the flag to 1, it indicates that the current architecture of the last-level cache needs to be changed.

[0018] In some possible implementations, when the LLC architecture flag is 0, it indicates that the current last-level cache will be switched to a shared last-level cache design.

[0019] When the LLC architecture flag is set to 1, it indicates that the current last-level cache will be switched to a private last-level cache design.

[0020] Secondly, a cache switching method for the aforementioned MCM-GPU adaptive final-level cache structure is also provided, the method comprising:

[0021] In response to the executed program, the LLC architecture change flag is read in real time during the program's execution, and it is determined whether the current last-level cache architecture needs to be changed.

[0022] If changes are needed, read the LLC schema flags.

[0023] Based on the LLC architecture flag bits read, switch the current last-level cache to either a private last-level cache design or a shared last-level cache design.

[0024] In some possible implementations, the method further includes:

[0025] Before reading the LL architecture flags, clear all cached data in the current last-level cache.

[0026] The main advantages of the technical solution of this invention are as follows:

[0027] The MCM-GPU adaptive last-level cache structure and its cache switching method of the present invention, by setting an adaptive last-level cache hardware structure including a local memory access queue, a remote memory access queue, an LLC architecture change flag register, and an LLC architecture flag register, can support dynamic switching configuration of shared last-level cache and private last-level cache. It can adaptively select the appropriate last-level cache architecture organization mode according to the program runtime configuration, meet various program memory access requirements, and improve the performance of MCM-GPU. Attached Figure Description

[0028] The accompanying drawings, which are included to provide a further understanding of embodiments of the invention and constitute a part of this invention, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0029] Figure 1 A schematic diagram of an MCM-GPU structure is provided as an example;

[0030] Figure 2 This is a schematic diagram of the MCM-GPU adaptive final-level cache structure according to an embodiment of the present invention;

[0031] Figure 3 This is a flowchart of a cache switching method according to an embodiment of the present invention;

[0032] Figure 4 This is a flowchart of another cache switching method according to an embodiment of the present invention. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0034] The technical solutions provided by the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0035] refer to Figure 2In a first aspect, an embodiment of the present invention provides an MCM-GPU adaptive final-level cache structure, which is configured in the GPU module of the MCM-GPU and includes: a Tag Array and a Date Array, wherein the Date Array is used to store data and the Tag Array is used to check whether the data corresponding to the address is in the cache, and further includes:

[0036] The local memory access queue is used to store memory access requests for the current GPU module.

[0037] The remote memory access queue is used to store memory access requests from other GPU modules in the MCM-GPU.

[0038] LLC architecture change flag register is used to store LLC architecture change flags that indicate whether the current architecture organization of the last-level cache needs to be changed.

[0039] The LLC architecture flag register stores LLC architecture flags that indicate whether the current last-level cache should be switched to a private last-level cache design or a shared last-level cache design when the architecture organization of the last-level cache needs to be changed.

[0040] in, Figure 2 Other components of the GPU module are also shown, including the stream processor SM and the on-chip interconnect network NoC.

[0041] Specifically, in one embodiment of the present invention, the MCM-GPU adaptive last-level cache structure, in response to the program executed in the MCM-GPU, reads the LLC architecture change flag bit in the LLC architecture change flag register in real time during program execution, and determines whether the current last-level cache architecture organization needs to be changed based on the LLC architecture change flag bit. If the LLC architecture change flag bit indicates that no change is needed, the program continues to execute based on the current last-level cache architecture organization. If the LLC architecture change flag bit indicates that a change is needed, the LLC architecture flag bit in the LLC architecture flag register is read. If the LLC architecture flag bit indicates that the current last-level cache should be switched to a private last-level cache design, then the current last-level cache should be switched to a private last-level cache design. If the LLC architecture flag bit indicates that the current last-level cache should be switched to a shared last-level cache design, then the current last-level cache should be switched to a shared last-level cache design. Based on the switched last-level cache architecture organization, the program continues to execute.

[0042] Furthermore, in one embodiment of the present invention, when the last-level cache uses a private last-level cache design, the memory access request issued by the stream processor in the current GPU module reaches the last-level cache via the on-chip interconnect network and is placed in the local memory access queue, waiting to access the last-level cache. When the last-level cache uses a shared last-level cache design, based on the memory space accessed by the memory address of the memory access request issued by the stream processor in the current GPU module, the memory access request is placed in the local memory access queue of the current GPU module or the remote memory access queue of another GPU module, waiting to access the last-level high-level cache; if the memory space accessed by the memory address of the memory access request issued by the stream processor in the current GPU module is local memory space, the memory access request is placed in the local memory access queue of the current GPU module; if the memory space accessed by the memory address of the memory access request issued by the stream processor in the current GPU module is remote memory space, the memory access request is placed in the remote memory access queue of the GPU module corresponding to the remote memory space.

[0043] Specifically, when the final level cache uses a private final level cache design, memory access requests issued by the stream processors in the current GPU module are placed in the local memory access queue, waiting to access the LLC. If the LLC is hit, the data is returned directly. If the LLC is missed, the memory access address of the memory access request determines whether to access local or remote storage space. If it is local storage space, the memory access request is sent directly to the memory connected to the current GPU module, and the data is returned to the SM that initiated the memory access request. If it is remote storage space, the memory access request is sent to the memory connected to the corresponding GPU module through the inter-GPU module network, and the data is returned to the SM that initiated the memory access request.

[0044] When the final level cache uses a shared final level cache design, the memory address accessed by the memory access request issued by the stream processor in the current GPU module is determined. If it is local memory, the memory access request is placed in the local memory access queue, waiting to access the LLC. If the LLC is hit, data is returned to the SM that initiated the memory access request. If the LLC is missed, the memory access request is sent to the memory connected to the current GPU module, and the data is returned to the SM that initiated the memory access request. If it is remote memory, the memory access request is sent to the corresponding GPU module through the inter-GPU module network and placed in the remote memory access queue of that GPU module, waiting to access the LLC of that GPU module. If the LLC is hit, data is returned to the SM that initiated the memory access request. If the LLC is missed, the memory access request is sent to the memory connected to that GPU module, and the data is returned to the SM that initiated the memory access request.

[0045] Furthermore, considering that registers can only store binary code, in one embodiment of the present invention, when the LLC architecture change flag is 0, it indicates that the current architecture organization of the last-level cache does not need to be changed; when the LLC architecture change flag is 1, it indicates that the current architecture organization of the last-level cache needs to be changed. Alternatively, it can be set as follows: when the LLC architecture change flag is 0, it indicates that the current architecture organization of the last-level cache needs to be changed; when the LLC architecture change flag is 1, it indicates that the current architecture organization of the last-level cache does not need to be changed.

[0046] Similarly, considering that registers can only store binary code, in one embodiment of the present invention, when the LLC architecture flag bit is 0, it indicates that the current last-level cache is switched to a shared last-level cache design; when the LLC architecture flag bit is 1, it indicates that the current last-level cache is switched to a private last-level cache design. Alternatively, it can be set as follows: when the LLC architecture flag bit is 0, it indicates that the current last-level cache is switched to a private last-level cache design; when the LLC architecture flag bit is 1, it indicates that the current last-level cache is switched to a shared last-level cache design.

[0047] refer to Figure 3 Secondly, an embodiment of the present invention also provides a cache switching method for the above-described MCM-GPU adaptive final-level cache structure, the method comprising the following steps:

[0048] In response to the executed program, the LLC architecture change flag is read in real time during the program's execution, and it is determined whether the current last-level cache architecture needs to be changed.

[0049] If changes are needed, read the LLC schema flags.

[0050] Based on the LLC architecture flag bits read, switch the current last-level cache to either a private last-level cache design or a shared last-level cache design.

[0051] Specifically, in response to the program executing in the MCM-GPU, the LLC architecture change flag bit in the LLC architecture change flag register is read in real time during program execution. Based on the LLC architecture change flag bit, it is determined whether the current last-level cache architecture needs to be changed. If the LLC architecture change flag bit indicates that no change is needed, the program continues to execute based on the current last-level cache architecture. If the LLC architecture change flag bit indicates that a change is needed, the LLC architecture flag bit in the LLC architecture flag register is read. If the LLC architecture flag bit indicates that the current last-level cache should be switched to a private last-level cache design, then the current last-level cache should be switched to a private last-level cache design. If the LLC architecture flag bit indicates that the current last-level cache should be switched to a shared last-level cache design, then the current last-level cache should be switched to a shared last-level cache design. Based on the switched last-level cache architecture, the program continues to execute.

[0052] Further, refer to Figure 4 In one embodiment of the present invention, the method further includes: before reading the LL architecture flag bit, clearing all cached data in the current last-level cache, that is, flushing all cached data in the current last-level cache.

[0053] By clearing all cached data in the current last-level cache before reading the LL architecture flags, the correctness of the program can be guaranteed when the last-level cache architecture organization method is dynamically switched.

[0054] An embodiment of the present invention provides an MCM-GPU adaptive last-level cache structure and its cache switching method. By setting an adaptive last-level cache hardware structure including a local memory access queue, a remote memory access queue, an LLC architecture change flag register, and an LLC architecture flag register, it can support dynamic switching configuration between shared last-level cache and private last-level cache. It can adaptively select the appropriate last-level cache architecture organization mode according to the program's runtime configuration, meet various program memory access requirements, and improve the performance of MCM-GPU.

[0055] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Additionally, the terms "front," "back," "left," "right," "upper," and "lower" in this document refer to the placement shown in the accompanying drawings.

[0056] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An MCM-GPU adaptive final-level cache structure, configured in the GPU module of an MCM-GPU, comprising: TagArray and Date Array, the Date Array is used to store data, and the Tag Array is used to check whether the data corresponding to the address is in the cache. The feature is that it also includes: A local memory access queue, which is used to store the memory access requests of the current GPU module; A remote memory access queue, which is used to store memory access requests from other GPU modules in the MCM-GPU; LLC architecture change flag register is used to store LLC architecture change flags that indicate whether the current architecture organization of the last-level cache needs to be changed. LLC architecture flag register is used to store LLC architecture flags that indicate whether the current last-level cache should be switched to a private last-level cache design or a shared last-level cache design when the architecture organization of the last-level cache needs to be changed. When the last-level cache uses a private last-level cache design, the memory access requests issued by the stream processors in the current GPU module are placed in the local memory access queue, waiting to access the last-level cache. When the last-level cache uses a shared last-level cache design, the memory access request is placed in the local memory access queue of the current GPU module or the remote memory access queue of other GPU modules according to the memory access address of the memory access request issued by the stream processor in the current GPU module, waiting to access the last-level high-level cache. If the memory access address of the memory access request issued by the stream processor in the current GPU module accesses the local memory space, the memory access request is placed in the local memory access queue of the current GPU module. If the memory access address of the memory access request issued by the stream processor in the current GPU module is a remote memory space, the memory access request is placed in the remote memory access queue of the GPU module corresponding to the remote memory space.

2. The MCM-GPU adaptive final-level cache structure according to claim 1, characterized in that, When the LLC architecture changes the flag to 0, it means that the current architecture of the last-level cache does not need to be changed. When the LLC architecture changes the flag to 1, it indicates that the current architecture of the last-level cache needs to be changed.

3. The MCM-GPU adaptive final-level cache structure according to claim 1, characterized in that, When the LLC architecture flag is 0, it indicates that the current last-level cache will be switched to a shared last-level cache design. When the LLC architecture flag is set to 1, it indicates that the current last-level cache will be switched to a private last-level cache design.

4. A cache switching method for the MCM-GPU adaptive final-level cache structure as described in any one of claims 1-3, characterized in that, include: In response to the executed program, the LLC architecture change flag is read in real time during the program's execution, and it is determined whether the current last-level cache architecture needs to be changed. If changes are needed, read the LLC schema flags. Based on the LLC architecture flag bits read, switch the current last-level cache to either a private last-level cache design or a shared last-level cache design.

5. The cache switching method according to claim 4, characterized in that, Also includes: Before reading the LL architecture flags, clear all cached data in the current last-level cache.

Images