A GPU shared l1 cache based on a polyhedral tag array

Abstract

The application discloses a GPU shared L1 cache based on an aggregated tag array. The main purpose of the application is to optimize the GPU shared L1 cache in the way of the aggregated tag array, so that the GPU shared L1 cache reduces the resource contention of the L1 cache, improves the parallelism of the GPU cache processing memory access requests, and thus improves the performance of the GPU shared L1 cache while fully utilizing the inter-core locality. Compared with the same type of GPU shared cache optimization method, the application greatly reduces the resource contention of the L1 cache due to sharing while not increasing the L2 cache memory access critical path.

Description

Technical fields:

[0001] This invention discloses a GPU shared L1 cache based on an aggregated tag array, which relates to GPU cache architecture, and particularly to GPU shared L1 cache architecture, belonging to the field of computer technology. Background technology:

[0002] GPUs, due to their massive parallelism and high energy efficiency, are widely used in various applications such as machine learning and high-performance computing. Caching plays a crucial role in GPU architecture design, as it addresses the memory wall problem. In a typical GPU two-level cache architecture, each GPU core has a private L1 cache, while all GPU cores share a single L2 cache. However, when processing applications with high inter-core locality, multiple GPU cores frequently access the same cache line. Data on the same cache line is loaded into the private L1 caches of different GPU cores, leading to data redundancy and impacting GPU cache efficiency.

[0003] Private L1 caches within shared GPU cores can eliminate data redundancy caused by multiple GPU cores accessing the same data, thus improving GPU cache utilization. Existing shared L1 caches can be divided into remote shared L1 caches and decoupled shared L1 caches.

[0004] (1) Remote Shared L1 Cache. In the remote shared L1 cache design, each L1 cache corresponds one-to-one with a GPU core and is mapped to the entire address space. Each L1 cache is not only connected to the L2 cache via the on-chip network, but also interconnected with other L1 caches via the on-chip network. When a request in one L1 cache misses, the metadata of the request is sent to other L1 caches via the on-chip network before accessing the L2 cache. If other L1 caches hit, the data is retrieved from the other L1 caches and returned to the GPU core; otherwise, the request is sent to the L2 cache. Although the remote shared cache can utilize data in other L1 caches, it increases the critical path for L2 cache access. In addition, data requests between L1 caches increase on-chip network latency, thus affecting GPU cache performance.

[0005] (2) Decoupled Shared L1 Cache. In the decoupled shared L1 cache design, each L1 cache is no longer one-to-one with a GPU core. Each L1 cache corresponds to a small segment of the address space, and all L1 caches are mapped together into the entire address space. When different GPU cores access the same address, they will be mapped to the same L1 cache, so there is no data redundancy between L1 caches. However, decoupled shared L1 cache can lead to multiple GPU cores accessing the same L1 cache, increasing L1 cache resource contention, serializing parallel memory access requests in the GPU, and resulting in low efficiency of the GPU cache.

[0006] In summary, to address the inefficiency of GPU caching in applications with high inter-core locality, existing GPU shared L1 caches are divided into remote shared L1 caches and decoupled shared L1 caches. Remote shared L1 caches increase the critical path for L2 cache access, and requests between L1 caches lead to increased latency in the on-chip network. In decoupled shared L1 caches, when multiple GPU cores access the same data simultaneously, parallel GPU memory access requests become serialized, impacting GPU cache performance. Therefore, existing GPU shared L1 caches still face inefficiency issues. Summary of the Invention:

[0007] This invention discloses a GPU shared L1 cache based on an aggregated tag array, comprising five parts: a GPU core decoupled from the L1 cache; an aggregated L1 cache tag array for storing data tags of the data array; a request dispatcher for distributing requests to a specified L1 cache data array; an L1 cache data array for storing data in the L1 cache; and an on-chip network for interconnecting the above components. The main objective of this invention is to optimize the GPU shared L1 cache based on an aggregated tag array, enabling the GPU shared L1 cache to fully utilize inter-core locality while reducing L1 cache resource contention, improving the parallelism of GPU cache in handling memory access requests, and thus enhancing the performance of the GPU shared L1 cache.

[0008] The basic process of requesting memory access using the GPU shared L1 cache based on the aggregated tag array is as follows: After the GPU core issues a request, the request first enters the aggregated tag array. Tag comparison is performed in the aggregated tag array, and the location of the requested memory access data in the data array is determined after the comparison. Then, the request reaches the request dispatcher, which determines the target data array. After reaching the data array (local data array) corresponding to the source GPU core, the request accesses the target data array through the on-chip network between the data arrays. After completing the access, the request returns to the local data array and, through the connection between the local data array and the GPU core, returns to the GPU core.

[0009] In the GPU shared L1 cache based on the aggregated tag array, while keeping other components in the GPU core unchanged, the L1 cache and the miss register are moved from inside the GPU core to outside the GPU core. At this time, the GPU core is the GPU core after being decoupled from the L1 cache.

[0010] In the GPU shared L1 cache based on aggregated tag arrays, the tag array and data array in the L1 cache are decoupled. The tag arrays obtained after decoupling multiple L1 caches are aggregated together and called the aggregated L1 cache tag array. The remaining portion of each L1 cache after removing the tag array is called the data array.

[0011] In the aggregated L1 cache tag array, each cache group's tag array resides in a separate cache memory to reduce cache memory conflicts. Each tag in each tag array is connected to a comparator after passing through a tag selector. An aggregated tag array formed by aggregating N tag arrays can process memory access requests from up to N GPU cores in parallel. After each request determines its group number through address resolution, it inputs the group number into the tag selector, which selects the cache tag for the corresponding group from all tag arrays and passes the cache tag to the tag comparator. The address tag in the request is compared with the cache tag in the tag comparator. If they match and the valid bit is true, it indicates a cache hit; otherwise, it indicates a cache miss. A key feature of the aggregated L1 cache tag array is its ability to support multiple memory access requests being compared with tags in multiple tag arrays in parallel. Therefore, requests originating from GPU cores can not only detect whether the requested data is cached in the local data array but also simultaneously detect whether the requested data is cached in other data arrays, thus accessing the data array where the requested data is already cached. The L1 cache data array is the same as the traditional L1 cache data array. It is used to cache data and can retrieve the correct memory access data by comparing the cache tags and sending it to the GPU core.

[0012] The request dispatcher selects the correct memory access target for a request after the request is compared with the tags in the aggregated tag array. After the request is compared with the tags in the aggregated tag array, the comparison result and request metadata reach the request dispatcher. The request dispatcher first determines whether the requested memory access data is in the current data array based on the tag comparison result. If it is, the current data array is accessed first. Otherwise, it determines whether the accessed data is in another L1 cache data array. If not, the L2 cache is accessed after passing through the miss register. If the requested memory access data is in another data array, to avoid large queuing delays caused by other data arrays being busy, the L2 cache is also accessed when other data arrays are busy. When other data arrays are idle, they are accessed.

[0013] The on-chip network has the following five functions:

[0014] (1) Port used to connect the GPU core and the aggregated tag array.

[0015] (2) Used to connect the aggregated tag array port to the request dispatcher.

[0016] (3) Used to connect the request dispatcher and the data array.

[0017] (4) Used to connect data arrays corresponding to different L1 caches within the same group.

[0018] (5) Used to connect the data array to the corresponding GPU core.

[0019] (6) Used to connect the data array and the L2 cache.

[0020] The advantages of this invention include:

[0021] This invention discloses a GPU shared L1 cache based on an aggregated tag array. Compared with existing designs, its advantages are: existing GPU shared caches face the problem of increasing the critical path for L2 cache access and severe L1 cache resource contention leading to low performance. This invention uses an aggregated tag array, which significantly reduces L1 cache resource contention due to sharing without increasing the critical path for L2 cache access, thereby improving the parallelism of GPU cache in handling memory access requests and ultimately enhancing the performance of the GPU shared L1 cache. Attached image description:

[0022] Figure 1 A schematic diagram of GPU shared L1 cache based on aggregated tag array.

[0023] Figure 2 Schematic diagram of aggregated tag array.

[0024] Figure 3 Request dispatch flowchart. Detailed implementation method:

[0025] The present invention will be further described in detail below with reference to the accompanying drawings.

[0026] like Figure 1 The diagram shows a shared L1 cache for GPUs based on an aggregated tag array. Figure 1The diagram showcases N GPU cores. Each GPU core's L1 cache is separated from the core. The tag array in the L1 cache is separate from the data array, and all tag arrays are aggregated together to form an aggregated tag array. Within the aggregated tag array, each smaller tag array still stores cached tags from the original L1 cache's data array. Each GPU core is connected to the aggregated tag array, which in turn is connected to the request dispatcher. The request dispatcher is connected to the data array, which is interconnected via an on-chip network. Furthermore, each data array is also connected one-to-one to a GPU core. Requests originating from the aggregated tag array, based on tag comparison results, first pass through the request dispatcher, which determines whether to access the local data array, proceed to another data array, or go to the L2 cache. For example, if GPU core 1 issues a request, the request first enters the aggregated tag array. There, the cached tag is compared with the memory access address tag, determining the location of the requested memory access data within the data array. Assuming the data requested is in data array 3, the request first arrives at request dispatcher 1 through the connection between the aggregated tag array and the request dispatcher (memory access requests from GPU core 1 are handled by request dispatcher 1). The request dispatcher selects the target data array, then passes through the on-chip network between L1 data arrays to access data array 3. After the access is completed, the request returns to data array 1 and returns to GPU core 1 through the connection between data array 1 and GPU core 1.

[0027] like Figure 2 The diagram shows a clustered tag array. Figure 2The diagram illustrates an aggregated tag array formed by two tag arrays. Tag array 1 and tag array 2 are decoupled from the L1 caches corresponding to GPU core 1 and GPU core 2, respectively, storing cache tags from data array 1 and data array 2. Each group of tag arrays resides in a separate cache memory to reduce cache memory conflicts. Each tag in each tag array is connected to a tag comparator after passing through a tag selector. The comparison result from the tag comparator is then processed by a comparison result processing unit and sent to the corresponding request dispatcher. For example, if request 1 from GPU core 1 accesses data with tag A, and request 2 from GPU core 2 accesses data with tag B, the cache group information is first parsed based on the memory access address of request 1. The tag of the corresponding group is then passed to the tag comparator and compared in parallel with request tag A. The comparison result is processed by the comparison result processing unit, which obtains the data array number where the data accessed by request 1 is located. The result is then sent to request dispatcher 1 to distribute the request to the target data array. Similarly, the final comparison result of request 2 from GPU core 2 is sent to request dispatcher 2. It is worth noting that requests 1 and 2 are processed in parallel. By aggregating the tag array, request tags from N GPU cores can be compared with N tag arrays in parallel, thereby reducing resource contention caused by L1 cache sharing.

[0028] like Figure 3 The diagram shows the request distribution process. Figure 3 The request dispatcher flowchart is shown. After the request is compared with the tags in the aggregated tag array, the comparison result and request metadata reach the request dispatcher. The request dispatcher first determines whether the data to be accessed for memory is in the current data array based on the tag comparison result. If it is, the current data array is accessed first. Otherwise, it determines whether the data to be accessed is in another L1 cache data array. If not, the L2 cache is accessed after passing through the miss register. If the data to be accessed for memory is in another data array, to avoid large queuing delays caused by other data arrays being busy, the L2 cache is also accessed when other data arrays are busy. When other data arrays are idle, they are accessed.

[0029] Finally, it should be noted that the present invention can also have many other applications. Without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding changes and modifications according to the present invention, but these corresponding changes and modifications should all fall within the protection scope of the present invention.

Claims

1. A GPU shared L1 cache device based on an aggregated tag array, characterized in that, It consists of the following components: a GPU core decoupled from the L1 cache; an aggregated L1 cache tag array for storing data tags for the data array; a request dispatcher for distributing requests to the specified L1 cache data array; an L1 cache data array for storing data in the L1 cache; and an on-chip network for interconnecting the above components. By optimizing the GPU shared L1 cache based on the aggregated tag array, the GPU shared L1 cache can fully utilize inter-core locality while reducing L1 cache resource contention, improving the parallelism of GPU cache in processing memory access requests, and thus improving the performance of the GPU shared L1 cache. The process of completing a memory access request using the GPU shared L1 cache based on the aggregated tag array includes: After the GPU core issues a request, the request first enters the aggregated tag array. In the aggregated tag array, tags are compared, and the location of the requested memory access data in the data array can be determined after the comparison. Then, the request reaches the request dispatcher, which determines the target data array. After the request reaches the local data array corresponding to the source GPU core, it accesses the target data array through the on-chip network between the data arrays. After completing the access, it returns to the local data array and returns to the GPU core through the connection between the local data array and the GPU core. In the GPU shared L1 cache based on aggregated tag arrays, the tag array and data array in the L1 cache are decoupled; the tag arrays obtained after decoupling multiple L1 caches are aggregated together and called aggregated L1 cache tag arrays; the remaining part of each L1 cache after removing the tag array is called the data array. In the aggregated L1 cache tag array, each cache group's tag array resides in a separate cache memory to reduce cache memory conflicts. Each tag in each tag array is connected to a comparator after passing through a tag selector. An aggregated tag array formed by aggregating N tag arrays can process memory access requests from up to N GPU cores in parallel. After each request determines its group number through address resolution, the group number is input to the tag selector, which selects the cache tag corresponding to the group from all tag arrays and passes the cache tag to the tag comparator. The address tag in the request is compared with the cache tag in the tag comparator; if they match... If the valid bit is true, it indicates a cache hit; otherwise, it indicates a cache miss. A key feature of the aggregated L1 cache tag array is that it supports multiple memory access requests to be compared with tags in multiple tag arrays in parallel. Therefore, requests originating from the GPU core can not only detect whether the requested data is cached in the local data array, but also simultaneously detect whether the requested data is cached in other data arrays, and then access the data array that has cached the requested data. The L1 cache data array is consistent with the traditional L1 cache data array, used to cache data, and can retrieve the correct memory access data through the cache tag comparison results and send it to the GPU core.

2. The apparatus according to claim 1, characterized in that: The request dispatcher is used to select the correct memory access target for the request after the request is compared with the tags in the aggregated tag array; after the request is compared with the tags in the aggregated tag array, the comparison result and the request metadata are sent to the request dispatcher. The request dispatcher first determines whether the data requested for memory access is in the current data array based on the tag comparison result. If it is, the current data array is accessed first. Conversely, it checks whether the accessed data is in the data array of other L1 caches. If not, it accesses the L2 cache after passing through the miss register. If the requested memory access data is on another data array, in order to avoid large queuing delays caused by other data arrays being busy, the L2 cache is also accessed when other data arrays are busy; when other data arrays are idle, the other data arrays are accessed.

Images