Method for sharing first-level cache memory by GPU (Graphics Processing Unit) through on-chip internet
By introducing Pseudo-Cache into the GPU, finding and forwarding missing memory fetch requests from L1 cache to other L1 caches that may contain hit data, the problem of GPU memory fetch latency is solved, and data sharing and GPU performance improvements between L1 caches are achieved.
Patent Information
- Application Number
- CN202411606770.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-12
- Publication Date
- 2025-03-04
AI Technical Summary
The memory fetch request delay caused by the L1 cache independence and L2 cache sharing of the GPU, especially when different SMs access the same data block, the data blocks are repeatedly stored in multiple L1 caches, increasing the memory fetch delay.
By introducing Pseudo-Cache, it is located between NoC and L2 cache, and saves the SM index information recently accessed from the DRAM side. When the L1 cache is missing, it is searched and forwarded to other L1 caches that may contain hit data through NoC, expanding the access scope of the L1 cache and realizing data sharing between the L1 cache.
Through data sharing between L1 caches, the average latency of memory fetch requests is reduced, the performance of the GPU is improved, and the pauses caused by the limitation of L2 cache resources are avoided.
Smart Images
Figure CN119557241A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of cache (high-speed cache memory) architecture in a GPU (graphics processing unit), and in particular to a method for GPUs to share a first-level high-speed cache memory through an on-chip interconnection network. Background Art
[0002] As a high-performance computing platform, GPU has been widely used in deep learning, graphics analysis, data analysis, computer-aided design and other fields. [1] . The hardware and software architecture of the GPU is very suitable for parallel computing. First, from a hardware perspective, the GPU is composed of dozens of independent SMs (stream processors), and each SM has dozens of cores for parallel computing. These SMs interact with DRAM (dynamic random access memory) through NoC (on-chip interconnect network). Secondly, from a software perspective, the application executed on the GPU is subdivided into several threads, and the threads are further organized into several thread blocks. The application assigns threads to each SM in units of thread blocks, and each SM is capable of processing thousands of threads simultaneously. Finally, combining hardware with software, the SM organizes threads into warps (thread bundles), where threads within a warp execute the same instructions in parallel on the SM, but multiple branches are allowed. This execution mode is called single instruction multiple threads (SIMT) [2] , making GPU programming more friendly.
[0003] In addition, in order to provide data acquisition services for GPU calculations more quickly, GPU uses high-bandwidth L1 cache (first-level cache memory) and L2 cache (second-level cache memory) [3][4] . Specifically, each L1 cache is private to the SM where it is located, and the address space it covers is the entire address space, which means that all L1 caches are independent of each other. The L2 cache is located between the NoC and DRAM and is shared by all SMs. The address space of each L2 cache is consistent with the address space of the DRAM channel connected to it, that is, all L2 caches are mapped to the entire address space together. Memory access requests that are missed in the L1 cache are sent to the L2 cache on the corresponding target DRAM channel side according to their addresses to obtain data. When the GPU performs task allocation, continuous thread blocks with locality may be emitted to different SMs. [5][6], so memory access requests from different SMs may access the same data block (or cache line) in the L2 cache. Ultimately, the same data block is stored repeatedly in multiple L1 caches. In addition, the latency of a memory access request that misses from the L1 cache to access the L2 cache is much longer than the latency of accessing the L1 cache. The reason is that, first, before accessing the L2 cache, the memory access request needs to go through more components to reach the L2 cache, such as: NoC and rasterization operation pipeline, which consumes hundreds of cycles. Second, when there are no resources available to process the memory access request in the L2 cache, the memory access request will stall, which directly affects the latency of getting data from the L2 cache.
[0004] References
[0005] [1]S.Shin, M.LeBeane, Y.Solihin, and A.Basu, "Neighborhood-aware addresstranslation for irregular gpu applications," in 2018 51st Annual IEEE / ACMInternational Symposium on Microarchitecture(MICRO).IEEE, 2018, pp.352-363
[0006] [2]NVIDIA CUDA C++Programming Guide,NVIDIA Corporation,2024
[0007] [3]J.Tan,K.Yan,SLSong,and
[0008] [4]H.Dai, C.Li, Z.Lin, and H.Zhou, "The demand for a sound baseline ingpu memory architecture research," in Proceedings of the Workshop onDuplicating, Deconstructing and Debunking(WDDD), 2017.
[0009] [5] M.Awatramani,
[0010] [6] M.Lee, S.Song, J.Moon, J.Kim, W.Seo, Y.Cho, and S.Ryu, "Improving gpgpuresource utilization through alternative thread block scheduling," in 2014IEEE20th international symposium on high performance computer architecture (HPCA). IEEE, 2014, pp.260-271. Summary of the invention
[0011] The present invention provides a method for GPU to share a first-level high-speed cache memory through an on-chip interconnection network. The present invention can reduce the average latency of memory access requests, improve the performance of GPU, and expand the accessible range of L1 cache by utilizing the repeated storage characteristics of data between L1 caches. See the following description for details:
[0012] A method for GPUs to share a primary cache memory via an on-chip interconnect network, the method comprising:
[0013] The Pseudo-Cache is located between the NoC and the L2 cache. The cache line of the Pseudo-Cache stores the SM index information of the cache line that was recently accessed from the DRAM side. When a read miss occurs in the L1 cache, the memory access request is sent to the target DRAM channel through the NoC.
[0014] When a memory access request reaches the Pseudo-Cache in the target DRAM channel, if a tag hit occurs, the SM index of the most recently accessed cache line is obtained from the hit cache line;
[0015] The memory access request is marked as a remote memory access request and is forwarded to the SM through the NoC according to the SM index information. If the memory access request can obtain data from the L1 cache of the SM, the memory access request is returned to the Pseudo-Cache through the NoC.
[0016] Then, it is sent to the source SM of the memory access request through the NoC, and the SM index of the cache line in the Pseudo-Cache is updated to the index of the source SM, which is used to indicate the SM that most recently accessed the data.
[0017] The method further includes: if a memory access request is missed in the remote SM, returning the memory access request to the Pseudo-Cache and sending it to the L2 cache.
[0018] Wherein, the method equips each memory access request queue between the L1 cache and the Pseudo-Cache with a dedicated buffer for storing remote memory access requests.
[0019] The method further includes: memory access requests of the remote buffer and the regular buffer in the same queue share the same data path, and the memory access requests in the two buffers use the data path in a polling order.
[0020] The method further comprises: maintaining the SM index field of the Pseudo-Cache cache line,
[0021] The cache line index that hits in the Pseudo-Cache is added to the remote memory access request;
[0022] When the Pseudo request is returned from the remote L1 cache, the cache line index carried by the remote memory access request is used to directly locate the original cache line position, and the tag of the remote memory access request is compared with the tag of the cache line to determine whether the original cache line is replaced;
[0023] If they are consistent, it means that they have not been replaced, otherwise it means that they have been replaced;
[0024] If the remote memory access request hits in the remote L1 cache and the cache line has been replaced in the Pseudo-Cache, a new cache line is found in the Pseudo-Cache to store the source SM index of the remote memory access request according to the replacement strategy;
[0025] If the remote memory access request hits in the remote L1 cache and the cache line has not been replaced in the Pseudo-Cache, the SM field of the cache line is updated to the source SM index of the memory access request;
[0026] If the remote memory access request in the remote L1 cache misses and the cache line in the Pseudo-Cache has been replaced, the memory access request is sent directly to the L2 cache;
[0027] If the remote memory access request in the remote L1 cache misses and the cache line in the Pseudo-Cache has not been replaced, the state of the cache line needs to be invalidated and then the memory access request is sent to the L2 cache.
[0028] The beneficial effects of the technical solution provided by the present invention are:
[0029] 1. The present invention utilizes the global view of the L2 cache end, and searches for other L1 caches that may contain hit data by accessing the Pseudo-Cache from the memory access request that has a miss in the L1 cache, and forwards the request through the on-chip interconnection network, thereby expanding the access range of the L1 cache and realizing data sharing between L1 caches;
[0030] 2. The present invention provides more data path options for memory access requests through data sharing between L1 caches, reduces the pause caused by limited L2 cache resources, thereby reducing the average delay of memory access requests and improving the performance of the GPU. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] Figure 1 A schematic diagram of a Pseudo-Cache (pseudo cache memory) architecture;
[0032] Figure 2 A schematic diagram of a data path for remote memory access requests;
[0033] Figure 3 This is the Pseudo-Cache workflow diagram. DETAILED DESCRIPTION
[0034] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention are described in further detail below.
[0035] The embodiment of the present invention proposes a Pseudo-Cache architecture. Figure 1 The overall architecture of a Pseudo-Cache connected to an L2 cache is shown. The key idea of Pseudo-Cache is to use the global insight of NoC on all L1 caches on the DRAM side to save the SM index of the cache line that was recently accessed from the DRAM side on the cache line of Pseudo-Cache, so that the miss request in L1 cache has the opportunity to track the duplicate data on other L1 caches through Pseudo-Cache. Specifically, when a read miss occurs in L1 cache, the memory access request is sent to the target DRAM channel through NoC. When the memory access request is output from NoC, it does not directly access the L2 cache, but first enters the Pseudo-Cache.
[0036] Unlike traditional cache lines used to store data, Pseudo-Cache only stores the SM index of the cache line that was most recently accessed, thereby reducing the hardware overhead of data storage. When a memory access request reaches the Pseudo-Cache, the tag is first checked. If a hit occurs, the SM index of the cache line that was most recently accessed is obtained from the hit cache line. Then, the memory access request is marked as a remote memory access request and forwarded to the SM through the NoC based on the SM index information. If the memory access request can successfully obtain data from the L1 cache of the SM, the memory access request is returned to the Pseudo-Cache through the NoC, and then sent to the source SM of the memory access request through the NoC, and the SM index of the cache line in the Pseudo-Cache is updated to the index of the source SM to indicate the SM that most recently accessed the data. If the memory access request is missed in the remote SM, the memory access request also needs to be returned to the Pseudo-Cache and further sent to the L2 cache. In addition, whether a memory access request hits or misses in the Pseudo-Cache, the corresponding memory access request information needs to be stored in the state holding register (SHR). For a hit memory access request, the SHR can reduce the number of requests to access the same cache line in the same remote L1 cache, thereby reducing the pressure on the NoC. For a miss request, the SHR plays the role of the traditional miss state holding register (MSHR). When data is returned from the L2 cache, the Pseudo-Cache also needs to be updated to record the SM that recently held this data.
[0037] Among them, the data path of Pseudo-Cache is as follows Figure 2As shown. In the traditional cache hierarchy, the regular memory access request generated from the L1 cache is sent to the L2 cache through the NoC, and then returns to the L1 cache to end. This is an open-loop process. However, after the introduction of the Pseudo-Cache, the remote memory access request generated from the Pseudo-Cache will be directed to the L1 cache instead of the L2 cache, and regardless of whether the remote memory access request is hit or missed in the L1 cache, it must be returned to the Pseudo-Cache. In the above process, if the regular memory access request and the remote memory access request share the memory access request queue resources, a closed loop will be formed. When there are no available resources in the memory access request queue, deadlock may occur. In order to eliminate potential deadlocks, the embodiment of the present invention is equipped with a dedicated buffer in each memory access request queue for storing remote memory access requests, and the original buffer is used to process regular memory access requests.
[0038] exist Figure 2 In the embodiment, the remote buffers in the first and fifth queues on the SM side of the NoC are used to store remote memory access requests that have completed access to the remote L1 cache, while the regular buffers in the same queue are used to store regular memory access requests that are missed in the L1 cache. The remote buffers in the sixth and fourth queues on the SM side of the NoC are used to store remote memory access requests forwarded by the Pseudo-Cache to the remote L1 cache for access, while the regular buffers in the same queue are used to store regular memory access requests returned to the L1 cache. The remote memory access requests and regular memory access requests in the first and fifth queues are respectively routed by the NoC to the remote buffer and regular buffer in the second queue on the DRAM side. Since the embodiment of the present invention is only designed for read requests, regular write requests will bypass the Pseudo-Cache and directly enter the L2cache. The return data of the remote memory access requests and regular memory access requests that need to be forwarded to the remote L1 cache are placed in the remote buffer and regular buffer in the third queue respectively. In order to simplify the data path, the memory access requests of the remote buffer and the regular buffer in the same queue share the same data path, and the memory access requests in the two buffers use the data path in a polling manner.
[0039] The embodiment of the present invention starts with a regular request to access the L1 cache to explain the specific implementation method. If the memory access request is missing in L1 cache 0, it is placed in the regular buffer in the first queue for storing regular memory access requests through the first demultiplexer, and then routed from the SM side by the NoC to the corresponding Pseudo-Cache side through the first multiplexer according to the memory access request address and address mapping relationship, and saved in the regular buffer in the second queue for storing regular memory access requests through the second demultiplexer. If the memory access request is hit in the Pseudo-Cache, the information of the memory access request is first stored in Figure 1 The SHR (status holding register) shown in the figure is used to merge requests to access the same cache line. Specifically, if a memory access request hits in the Pseudo-Cache, the SHR will further check the memory access request. If the memory access request hits in the SHR, it means that a memory access request to access the same cache line has been sent to the same remote L1 cache, so there is no need to send the memory access request to the remote L1 cache. If the memory access request does not find any hits in the SHR, the memory access request extracts the SM index from the hit Pseudo-Cache cache line, sets itself as a remote memory access request, and then places it in the remote buffer in the third queue for storing remote memory access requests. When the remote memory access request is selected by the second multiplexer in the remote buffer, the NoC forwards the remote memory access request to the L1 cache 1 in the corresponding SM based on the SM index information obtained from the Pseudo-Cache cache line. On the SM side, the remote memory access request is stored in the remote buffer in the fourth queue through the third demultiplexer, waiting to be selected by the third multiplexer to access the L1 cache 1 of the SM.
[0040] It should be noted that there is no consistency between the L1 cache and the Pseudo-Cache. The data predicted in the Pseudo-Cache to hit in the remote L1 cache may have been replaced by other local memory access requests. Therefore, even if it is predicted to hit in the Pseudo-Cache, the remote memory access request may be missed in the remote L1 cache. Regardless of whether the remote memory access request hits in the remote L1 cache 1, the remote memory access request and its hit status are saved in the remote buffer in the fifth queue for saving remote memory access requests through the fourth demultiplexer, and then sent back to the Pseudo-Cache side by the NoC through the fourth multiplexer. The remote memory access request returned from the remote L1 cache 1 is stored in the remote buffer in the second queue through the second demultiplexer. When the remote memory access request is responded to by the Pseudo-Cache, the remote memory access request tag is deleted and becomes a regular memory access request, and the Pseudo-Cache checks its hit status. If it hits in the remote L1 cache 1, its hit data will be placed in the regular buffer in the third queue, and then selected by the second multiplexer and sent back to the source SM side through the NoC, saved in the regular buffer in the sixth queue through the fifth demultiplexer, and finally saved in the L1 cache 0 after being selected by the fifth multiplexer. If it misses in the remote L1 cache 1, the request needs to access the L2 cache to obtain data as a regular memory access request. When the memory access request returns from the L2 cache, the memory access request selects a cache line in the Pseudo-Cache to save the tag and source SM index information according to the replacement policy.
[0041] In order to more accurately record the SM index of the most recently accessed Pseudo-Cache cache line, it is necessary to maintain the SM index field of the Pseudo-Cache cache line. When a remote memory access request in the remote L1 cache is missed, it means that the SM index information stored in the corresponding cache line of the Pseudo-Cache has expired, and the cache line needs to be set to an invalid state. In order to reduce the tag comparison required to find the cache line, the cache line index hit in the Pseudo-Cache is also added to the remote memory access request. When the remote memory access request is returned from the remote L1 cache, the original cache line corresponding to the remote memory access request in the Pseudo-Cache may have been replaced. At this time, the original cache line position is directly located through the cache line index carried by the remote memory access request, and then the tag of the remote memory access request and the tag of the cache line are compared to determine whether the original cache line has been replaced. If they are consistent, it means that it has not been replaced, otherwise it means that it has been replaced. If the remote memory access request hits in the remote L1cache and the cache line has been replaced in the Pseudo-Cache, a new cache line needs to be found in the Pseudo-Cache according to the replacement strategy to store the source SM index of the remote memory access request. If the remote memory access request hits in the remote L1cache and the cache line has not been replaced in the Pseudo-Cache, considering that the data is more likely to remain in the source SM of the memory access request for a longer time, at this time, only the SM field of the cache line needs to be updated to the source SM index of the memory access request. If the remote memory access request in the remote L1 cache does not hit and the cache line in the Pseudo-Cache has been replaced, this memory access request is sent directly to the L2 cache and no content of the Pseudo-Cache needs to be changed. If the remote memory access request in the remote L1 cache does not hit and the cache line in the Pseudo-Cache has not been replaced, the state of the cache line needs to be invalidated and then the memory access request is sent to the L2 cache. The workflow diagram of the Pseudo-Cache is shown as follows. Figure 3 shown.
[0042] Unless otherwise specified, the models of the components in the embodiments of the present invention are not limited, and any device that can perform the above functions may be used.
[0043] Those skilled in the art will appreciate that the accompanying drawing is only a schematic diagram of a preferred embodiment, and the serial numbers of the embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.
[0044] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A method for GPUs to share a primary cache memory via an on-chip interconnect network, characterized in that: The method comprises: The Pseudo-Cache is located between the NoC and the L2 cache. The cache line of the Pseudo-Cache stores the SM index information of the cache line that was recently accessed on the DRAM side. When a read miss occurs in the L1 cache, the memory access request is sent to the target DRAM channel through the NoC. When a memory access request reaches the Pseudo-Cache on the DRAM channel side, if a tag hit occurs, the SM index of the most recently accessed cache line is obtained from the hit cache line; The memory access request is marked as a remote memory access request and is forwarded to the SM through the NoC according to the SM index information. If the memory access request can obtain data from the L1 cache of the SM, the memory access request is returned to the Pseudo-Cache through the NoC. Then, it is sent to the source SM of the memory access request through the NoC, and the SM index of the cache line in the Pseudo-Cache is updated to the index of the source SM, which is used to indicate the SM that most recently accessed the data.
2. The method for GPUs to share a primary cache memory via an on-chip interconnect network according to claim 1, characterized in that: The method further includes: if a memory access request is missed in the remote SM, returning the memory access request to the Pseudo-Cache, and then sending the memory access request to the L2 cache.
3. The method for GPUs to share a primary cache memory via an on-chip interconnect network according to claim 1, characterized in that: The method equips each memory access request queue between the L1 cache and the Pseudo-Cache with a dedicated buffer for storing remote memory access requests.
4. The method for GPUs to share a primary cache memory via an on-chip interconnect network according to claim 1, characterized in that: The method further includes: memory access requests of the remote buffer and the regular buffer in the same queue share the same data path, and the memory access requests in the two buffers use the data path in a polling order.
5. The method for GPUs to share a primary cache memory via an on-chip interconnect network according to claim 1, characterized in that: The method further comprises: maintaining an SM index field of a Pseudo-Cache cache line, The cache line index that hits in the Pseudo-Cache is added to the remote memory access request; When the Pseudo request is returned from the remote L1 cache, the cache line index carried by the remote memory access request is used to directly locate the original cache line position, and the tag of the remote memory access request is compared with the tag of the cache line to determine whether the original cache line is replaced; If they are consistent, it means that they have not been replaced, otherwise it means that they have been replaced; If the remote memory access request hits in the remote L1 cache and the cache line has been replaced in the Pseudo-Cache, a new cache line is found in the Pseudo-Cache to store the source SM index of the remote memory access request according to the replacement strategy; If the remote memory access request hits in the remote L1 cache and the cache line has not been replaced in the Pseudo-Cache, the SM field of the cache line is updated to the source SM index of the memory access request; If the remote memory access request in the remote L1 cache misses and the cache line in the Pseudo-Cache has been replaced, the memory access request is sent directly to the L2 cache; If the remote memory access request in the remote L1 cache misses and the cache line in the Pseudo-Cache has not been replaced, the state of the cache line needs to be invalidated and then the memory access request is sent to the L2 cache.