Method and device for improving data transmission efficiency of GPU (Graphics Processing Unit) through bandwidth sharing

Abstract

The invention discloses a method and a device for improving data transmission efficiency by a GPU (Graphics Processing Unit) through bandwidth sharing, and the method comprises the following steps of: grouping L2 cache sub-partitions in a manner that four sub-partitions form a group, sharing a data port by the four sub-partitions in the group, and jointly executing a data return task of a single sub-partition; the L2 cache judges whether the current sub-partition is served or not, and the scheduler in the group checks the next sub-partition through polling scheduling and judges whether the current sub-partition has a request or not. If the current sub-partition has a request and the head-of-queue request is read hit, determining to serve the sub-partition; when the current sub-partition is not served and the queue head request is a write request or read missing, the sub-partition and the served sub-partition are subjected to parallel processing; the L2 cache divides the data into four parts and sends the four parts to the multiplexer module, and the multiplexer module of each sub-partition selects corresponding signal input and transmits the data to a data port; the aggregated data port sends the data back to the L1 cache through the NoC; and counting and confirming that the complete cache line data is received by a counter in the SM. The device comprises a processor and a memory.

Description

Technical Field

[0001] This invention relates to the field of data transmission bandwidth optimization in the storage architecture of GPUs (Graphics Processing Units), and more particularly to a method and apparatus for improving data path utilization and data transmission efficiency of GPUs by sharing bandwidth. Background Technology

[0002] GPUs, as a high-performance computing platform, have been widely used in fields such as deep learning, graphics analysis, data analysis, and computer-aided design. [1] GPUs have hardware and software architectures that are well-suited for parallel computing, and their architectures include... Figure 1 As shown. First, from a hardware perspective, a GPU consists of dozens of independent Stream Processors (SMs), each containing dozens of cores for parallel computing. These SMs interact with DRAM (Dynamic Random Access Memory) via NoC (NoC). Second, from a software perspective, applications executing on the GPU are subdivided into threads, which are further organized into thread blocks. The application allocates threads to various SMs in units of thread blocks, and each SM can handle thousands of threads simultaneously. Finally, combining hardware and software, the SMs organize threads into warps, where threads within a warp execute the same instructions in parallel on the SM, but are allowed to have multiple branches. This execution mode is called Single Instruction Multithreading (SIMT). [2] This makes GPU programming more user-friendly. To achieve further optimization, the GPU employs a warp-based memory access request merging mechanism. When the 32-bit data addresses accessed by all threads in a thread bundle fall within the same 128-byte aligned contiguous memory block, the hardware automatically merges these requests into a single global memory access operation. This merging mechanism significantly reduces the number of memory accesses, improving DRAM bandwidth utilization by up to several times.

[0003] In addition, to provide data acquisition services for GPU computing more quickly, the GPU uses high-bandwidth L1 cache (Level 1 cache) and L2 cache (Level 2 cache). [3][4]Specifically, each L1 cache is private to its respective SM (Streaming Service Controller), and its address space encompasses the entire address space, meaning all L1 caches are independent of each other. The L2 cache resides between the NoC (No-Channel Block) and DRAM, and is shared by all SMs. The address space of each L2 cache is consistent with the address space of the DRAM channel it is connected to; that is, all L2 caches are mapped to the entire address space. Memory access requests that miss in the L1 cache are sent to the corresponding L2 cache on the target DRAM channel side to retrieve the data based on its address. To support higher concurrency, each L2 cache in the GPU is further divided into multiple partitions, each corresponding to one DRAM channel, and each partition is further subdivided into two sub-partitions. Each sub-partition is equipped with an independent request queue, tag array, and a 32-byte / cycle data port. [5] It is used to handle memory requests initiated from the SM and data transfer tasks returned to the L1 cache. This sub-partition design improves parallel processing capabilities and reduces bank (memory) conflicts.

[0004] Both L1 and L2 caches use cache lines as the basic management unit, with each cache line being 128 bytes in size. Therefore, read requests from the L1 cache to the L2 cache are always 128-byte data requests. When the required data is hit in the L2 cache sub-partition, the target cache line data is returned to the L1 cache via the NoC (No-Cross-Cache), using the same port as the L2 cache sub-partition data port. The NoC bandwidth is also 32 bytes per cycle. [6] Therefore, each 128-byte cache line of data requires four consecutive cycles to complete the transfer. In high-concurrency application scenarios, requests initiated by multiple SMs may concentrate on accessing a few sub-partitions, causing these sub-partitions to become saturated while other sub-partitions remain idle. This results in a severe underutilization of the overall L2 cache bandwidth, and the fact that each memory access request requires four consecutive cycles to send back the data further exacerbates this problem.

[0005] References

[0006] [1]S. Shin, M. LeBeane, Y. Solihin, and A. Basu, “Neighborhood-awareaddress translation for irregular gpu applications,” in 2018 51st AnnualIEEE / ACM International Symposium on Microarchitecture (MICRO). IEEE, 2018,pp. 352–363

[0007] [2]NVIDIA CUDA C++ Programming Guide, NVIDIA Corporation, 2024

[0008] [3]J. Tan, K. Yan, S. L. Song, and X. Fu, “Loscache: Leveraginglocality similarity to build energy-efficient gpu l2 cache,” in 2019 Design,Automation & Test in Europe Conference & Exhibition (DATE). IEEE, 2019, pp.1190–1195.

[0009] [4]H. Dai, C. Li, Z. Lin, and H. Zhou, “The demand for a soundbaseline in gpu memory architecture research,” in Proceedings of the Workshopon Duplicating, Deconstructing and Debunking (WDDD), 2017.

[0010] [5] Zhao X, Adileh A, Yu Z, et al. Adaptive memory-side last-levelGPU caching[C] / / Proceedings of the 46th international symposium on computerarchitecture. 2019: 411-423.

[0011] [6] Ibrahim MA, Kayiran O, Eckert Y, et al. Analyzing and leveragingdecoupled L1 caches in GPUs[C] / / 2021 IEEE International Symposium on High-Performance Computer Architecture (HPCA). IEEE, 2021: 467-478. Summary of the Invention

[0012] This invention provides a method and apparatus for improving data transmission efficiency of a GPU through bandwidth sharing. By aggregating the data ports of sub-partitions, this invention accelerates the return speed of a single memory access request. Simultaneously, it utilizes the idle bandwidth of NoC (No-Card) to reduce the total number of processing cycles for the same number of memory access requests, thereby improving bandwidth utilization and ultimately enhancing GPU performance. See the description below for details:

[0013] A first aspect: a method for improving data transmission efficiency of a GPU through bandwidth sharing, the method comprising:

[0014] 1) Group the L2 cache sub-partitions into groups of four, with the four sub-partitions in each group sharing a data port and jointly executing the data back-to-data task for a single sub-partition;

[0015] 2) The L2 cache determines whether to serve the current sub-partition. The group scheduler checks the next sub-partition using round-robin scheduling to determine if there are any requests for the current sub-partition. If there are requests for the current sub-partition and the first request in the queue is a read hit, then this sub-partition is served. If the current sub-partition is not served and the first request in the queue is a write request or a read miss, then this sub-partition is processed in parallel with the sub-partition being served.

[0016] 3) The L2 cache divides the data into four parts and sends them to the multiplexer module. The multiplexer module of each sub-partition selects the corresponding signal input and transmits the data to the data port; the aggregated data port sends the data back to the L1 cache through NoC.

[0017] 4) The counter in SM confirms that complete cache line data has been received.

[0018] If there are no requests for the current subpartition and the request at the head of the queue is not a read hit, the scheduler will be re-executed to poll and check the next subpartition.

[0019] When serving the current sub-partition, the target cache line is found, and then step 3 is executed.

[0020] Specifically, the L2 cache determines whether to serve the current sub-partition. If it does not serve the current sub-partition and the request at the head of the queue is a write request or a read miss, then the sub-partition and the served sub-partition are processed in parallel.

[0021] The data port of the current sub-partition is requisitioned by the sub-partition that needs to be served to send back data; at the same time, if the request at the head of the queue of the current sub-partition is a write request, the data will be evicted to the corresponding cache line in DRAM; if it is a read miss request, the request will be sent to DRAM.

[0022] Specifically, all L2 cache subpartitions are grouped into groups of four:

[0023] A step-by-step grouping method is adopted, with each sub-partition within the group separated by a fixed step size. The grouping step size is the total number of sub-partitions divided by 4, which avoids local hotspots.

[0024] The method further includes:

[0025] The cache line data corresponding to the four sub-partitions in each group is split into four segments, each segment being 32 bytes, and sent to the four sub-partition multiplexer modules of that group. Each sub-partition multiplexer module selects one data output path based on the polling result as the selection control signal and sends the data to the data port. The data is then transmitted in parallel to the SM that issued the request through their respective NoC paths.

[0026] The SM includes a counter for tracking the reception of segmented data. When it is detected that all four segments of data in the cache line have been returned, the request is considered to be completed.

[0027] In a second aspect, an apparatus for improving data transmission efficiency of a GPU through bandwidth sharing, the apparatus comprising: a processor and a memory, the memory storing program instructions, the processor calling the program instructions stored in the memory; if the program instructions are memory access instructions and the generated memory access request requires access to the L2 cache, the apparatus performs the method described in any of the first aspects.

[0028] Third aspect, a computer-readable storage medium storing a computer program, the computer program including program instructions, wherein when the program instructions are executed by a processor, if the program instructions are memory access instructions and the generated memory access request requires access to the L2 cache, the processor performs the method described in any of the first aspects.

[0029] The beneficial effects of the technical solution provided by this invention are:

[0030] 1. This invention breaks the limitation of physical sub-partition independence by logically aggregating the data ports of sub-partitions, combining the scattered ports of four sub-partitions into a logically unified 128-byte / cycle high-bandwidth channel, improving bandwidth utilization, and avoiding the concentration of access hotspots in the same physical area through a step-based grouping strategy.

[0031] 2. This invention ensures balanced processing of requests from all sub-partitions through a round-robin scheduling mechanism, thereby improving the overall stability and throughput of the system. This enables the GPU L2 cache to achieve significant performance improvement in high-concurrency scenarios and reduces network congestion in practical applications. Attached Figure Description

[0032] Figure 1 This is a schematic diagram of the traditional GPU architecture;

[0033] Figure 2 A schematic diagram of the bandwidth sharing architecture provided by the present invention;

[0034] Figure 3 A flowchart illustrating the operation of the bandwidth sharing architecture provided by this invention;

[0035] Figure 4 A comparison diagram of the memory access request pipelines for traditional architecture and bandwidth-shared architecture. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below.

[0037] Example 1

[0038] One method for improving data transfer efficiency in GPUs through bandwidth sharing, see [link to relevant documentation]. Figure 2 and Figure 3 The method includes:

[0039] This invention leverages the uneven utilization of GPU resources by sharing a data path to accelerate data transfer rates using idle resources. To achieve bandwidth sharing, this invention departs from the traditional architecture where sub-partitions operate independently and process memory access requests in parallel. Instead, it groups all L2 cache sub-partitions into groups of four, using a round-robin scheduling method to process read hit requests from individual sub-partitions sequentially in each cycle. The 32-byte bandwidth data ports of each group of four sub-partitions are logically aggregated to form a 128-byte / cycle virtual channel, enabling cache line data to be sent to the NoC for transmission within a single cycle. A schematic diagram of the architecture of this invention is shown below. Figure 2 As shown.

[0040] To fully utilize idle bandwidth resources, this embodiment of the invention further employs a step-by-step grouping strategy. Unlike continuous grouping, this approach effectively avoids the risk of multiple spatially adjacent requests clustering in the same group, thereby balancing access load and improving system stability. For write requests and read miss requests (these two types of requests do not require occupying data ports), sub-partitions can process them in parallel, sending write data or requests to DRAM without affecting the data port's transmission of read hit data to other sub-partitions.

[0041] Figure 3 This document illustrates the overall memory access request processing flowchart of an embodiment of the present invention. During execution, when the L1 cache sends a 128-byte data request to the L2 cache, the scheduler within the group selects a sub-partition with a processable read hit request from the current group according to the group's polling order. The cache line data corresponding to this sub-partition is split into four segments, each 32 bytes, and sent to the MUX (Multiplexer) modules of the four sub-partitions in the group. Each sub-partition's MUX module sends the data to the data port and then transmits it in parallel to the requesting SM through its respective NoC path, thus completing the sending task that originally required four cycles in one cycle. A 2-bit counting logic is added to each L1 cache to track the return status of each data segment. Once all four data segments have been received, it is considered that the data required by the L1 cache has been received, and the cache line request processing is complete.

[0042] During the polling process within a group, the scheduler has the ability to judge and skip: when a sub-partition's request queue is empty, or the current head request is a write operation or a read miss, the scheduler will automatically skip that sub-partition and try the next one until it finds a sub-partition with a read hit request or completes the traversal of all sub-partitions in the group, avoiding idle data ports and improving scheduling efficiency. Meanwhile, if the skipped sub-partition's current request does not require occupying a data port, it can independently send write data to DRAM or forward read miss requests to DRAM within the same cycle, maximizing the parallel processing capability of each sub-partition.

[0043] Example 2

[0044] The following will be combined with specific examples Figure 2 and Figure 3 The formula further describes the scheme in Example 1, as detailed below:

[0045] This invention proposes a bandwidth sharing architecture. Figure 2This paper illustrates the specific structure within a single group after sub-partition grouping is completed under the architecture of this invention embodiment. Based on the traditional GPU architecture, this invention embodiment adds a MUX between the cache line area and data port of each sub-partition, connecting the data ports of each sub-partition within the group to other sub-partitions to form a lightweight network, achieving logical aggregation of sub-partitions. To implement intra-group round-robin, a scheduler is added within each sub-partition group to select the sub-partition to be served in each cycle. Additionally, on the SM side, a 2-bit logical counter is added to the L1 cache for each missing read request to track whether the requested data has been collected. The key idea of ​​bandwidth sharing in this invention embodiment is to utilize the underutilization of L2 cache sub-partitions and NoC bandwidth in the GPU, improving data transmission efficiency through idle data paths.

[0046] Specifically, in this embodiment of the invention, all L2 cache sub-partitions are logically grouped into groups of four, and the four data ports within each group are logically aggregated to jointly transmit data returned to the L1 cache. To improve the balance of request distribution, this embodiment of the invention employs a step-by-step grouping strategy.

[0047] For example, in a system containing 24 sub-partitions, the partitions are divided with a step size of 6: sub-partitions numbered 0, 6, 12, and 18 form one group, sub-partitions numbered 1, 7, 13, and 19 form another group, and so on, ultimately forming 6 groups. The specific formula for calculating the grouping step size is as follows:

[0048]

[0049] After grouping is completed, when a request from the L1 cache hits the L2 cache, the target 128-byte cache line will be split into four 32-byte segments of data, which will be sent out through the four sub-partition data ports within the group, and each segment will be transmitted in parallel to the SM that initiated the request through the NoC path.

[0050] Regarding the scheduling mechanism, this embodiment of the invention employs round-robin scheduling. In each cycle, the scheduler sequentially traverses the sub-partitions within each group, selecting sub-partitions that meet the conditions for service. During the traversal, the scheduler will skip the following two types of sub-partitions: (1) those with empty request queues; and (2) those with non-empty request queues but whose head request is a write request or a missing read request, as these two types of requests do not require data ports. In this way, the scheduler can effectively avoid empty round-robin and improve scheduling efficiency.

[0051] Once the scheduler determines the service sub-partition for the current period, that sub-partition begins processing the normal read hit request at the head of its queue. After locating the target cache line for the request, this embodiment of the invention divides the cache line into four 32-byte segments and sends them to the MUX modules before the data ports of each sub-partition within the group. The MUX modules use the polling result as a selection control signal to select one data output path and send the data to the data port. Each path then sends the data to the NoC and returns to the SM that issued the request. The receiving logic within the SM includes a counter for tracking the reception of segmented data. When it is detected that all four segments of the cache line have been returned, the request is considered complete.

[0052] Example 3

[0053] The following is combined Figure 4 The feasibility of the schemes in Examples 1 and 2 is verified, as detailed below:

[0054] Figure 4 This diagram illustrates a comparison of the processes of a traditional architecture and a bandwidth-shared architecture in handling memory requests under a simulated memory access request processing scenario. The diagram shows the pipeline for L2 cache request processing, data port access, and NoC data retransmission in each cycle. It assumes requests ①, ④, and ⑬ originate from SM 0, ②, ③, ⑤, and ⑩ from SM 1, ⑥, ⑦, ⑧, and ⑪ from SM 2, and ⑨ and ⑫ from SM 3. Figure 4 The upper part illustrates the request processing flow under a traditional data transmission architecture. Each sub-partition processes requests independently, acquires data, and transmits it to the NoC via a data port. The NoC then returns the data to the SM, allowing multiple sub-partitions to execute in parallel. Each data transmission takes four cycles. In this independent sub-partition architecture, uneven request volumes can lead to uneven utilization. For example, if sub-partition 12 receives fewer requests, it remains idle until other sub-partitions have finished processing, resulting in underutilization of its data port and corresponding NoC channel. Figure 4The lower half illustrates the request processing flow under a bandwidth-shared architecture. In cycle 1, sub-partition 0 processes its read hit request. Simultaneously, sub-partition 6 receives a miss request, which does not require its data port, thus allowing it to independently complete memory access and forward the request to DRAM. In cycle 2, the scheduler rounds to sub-partition 6, but the head request of this sub-partition is a write hit, also requiring no data port. Therefore, the scheduler jumps to sub-partition 12, and sub-partition 6 directly sends the write data to DRAM. Similarly, in cycles 4 and 6, sub-partitions 6 and 18 process read misses or write requests respectively, neither occupying a data port. In this simulated scenario of uneven request distribution, this embodiment of the invention significantly improves the processing efficiency of the L2 cache. Although transferring data from the L2 cache back to the SM via NoC still requires 4 cycles, the overall processing cycle count can still be reduced due to the interleaved transmission of various requests, ultimately saving 3 cycles.

[0055] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0056] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for improving data transmission efficiency of a GPU through bandwidth sharing, characterized in that, The method includes: 1) Group the L2 cache sub-partitions into groups of four, with the four sub-partitions in each group sharing a data port and jointly executing the data back-to-data task for a single sub-partition; 2) The L2 cache determines whether to serve the current sub-partition. The group scheduler checks the next sub-partition using round-robin scheduling to determine if there are any requests for the current sub-partition. If there are requests for the current sub-partition and the first request in the queue is a read hit, then this sub-partition is served. If the current sub-partition is not served and the first request in the queue is a write request or a read miss, then this sub-partition is processed in parallel with the sub-partition being served. 3) The L2 cache divides the data into four parts and sends them to the multiplexer module. The multiplexer module of each sub-partition selects the corresponding signal input and transmits the data to the data port; the aggregated data port sends the data back to the L1 cache through NoC. 4) The counter in SM confirms that complete cache line data has been received.

2. The method for improving data transmission efficiency of a GPU through bandwidth sharing according to claim 1, characterized in that, If there are no requests for the current subpartition and the request at the head of the queue is not a read hit, the scheduler will be re-executed to poll the next subpartition.

3. The method for improving data transmission efficiency of a GPU through bandwidth sharing according to claim 1, characterized in that, When the service is in the current sub-partition, the target cache line is found, and then step 3 is executed.

4. A method for improving data transmission efficiency of a GPU through bandwidth sharing according to claim 1, characterized in that, The L2 cache determines whether to serve the current sub-partition. If it does not serve the current sub-partition and the request at the head of the queue is a write request or a read miss, then the sub-partition and the served sub-partition are processed in parallel, specifically: The data port of the current sub-partition is requisitioned by the sub-partition that needs to be served to send back data; at the same time, if the request at the head of the queue of the current sub-partition is a write request, the data write will be evicted to the corresponding cache line in DRAM. If it is a read missing request, the request is sent to DRAM.

5. A method for improving data transmission efficiency of a GPU through bandwidth sharing according to claim 4, characterized in that, All L2 cache subpartitions are grouped into groups of four, specifically: A step-by-step grouping method is adopted, with each sub-partition within the group separated by a fixed step size. The grouping step size is the total number of sub-partitions divided by 4, which avoids local hotspots.

6. A method for improving data transmission efficiency of a GPU through bandwidth sharing according to claim 4, characterized in that, The method further includes: The cache line data corresponding to the four sub-partitions in each group is split into four segments, each segment being 32 bytes, and sent to the four sub-partition multiplexer modules of that group. Each sub-partition multiplexer module selects one data output path based on the polling result as the selection control signal and sends the data to the data port. The data is then transmitted in parallel to the SM that issued the request through their respective NoC paths.

7. A method for improving data transmission efficiency of a GPU through bandwidth sharing according to claim 1, characterized in that, The SM includes a counter for tracking the reception of segmented data. When it is detected that all four segments of data in the cache line have been returned, the request is considered to be completed.

8. A device for improving data transmission efficiency of a GPU through bandwidth sharing, characterized in that, The apparatus includes a processor and a memory, the memory storing program instructions, the processor calling the program instructions stored in the memory; if the program instructions are memory access instructions and the generated memory access request requires access to the L2 cache, the apparatus performs the method described in any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions, and when the program instructions are executed by a processor, if the program instructions are memory access instructions and the generated memory access request requires access to the L2 cache, then the processor performs the method described in any one of claims 1-7.

Images