Stochastic Optimization of Surface Cacheability on Parallel Processing Units

Abstract

A processing system [100] selectively allocates storage in a local cache [120] of a parallel processing unit [110] to cache lines of a repeating pattern of data [420] that exceed the storage capacity of the cache. The processing system identifies a repeating pattern of data having cache lines with a reuse distance that exceeds the storage capacity of the cache. A cache controller [130] allocates storage in the cache to only a subset [140] of the cache lines of the repeating pattern of data and excludes the remainder [415] of the cache lines of the repeating pattern of data from the cache. By restricting the cache to store only the subset of the cache lines of the repeating pattern of data, the cache controller increases the hit rate in the cache for the subset of cache lines.

Description

[Background technology]

[0001] Processing systems that include parallel processing units, such as graphics processing units (GPUs), implement cache hierarchies (or multilevel caches) that use a hierarchy of caches of varying speeds to store frequently accessed data. More frequently requested data is typically cached in a relatively fast cache (e.g., L1 cache) that is physically (or logically) located closer to the processor core or compute unit. Higher levels of cache (e.g., L2 cache, L3 cache) store less frequently requested data. The last-level cache (LLC) is the highest-level (and lowest-access speed) cache; it reads and writes data directly from and to system memory. Caches differ from system memory because they implement a cache replacement policy that replaces data within a cache line as new data needs to be written to the cache line. For example, a least-recently-used (LRU) policy replaces data within the cache line that has not been accessed for the longest time by evicting the data in the LRU cache line and writing new data to the LRU cache line. Summary of the Invention [Means for solving the problem]

[0002] One example includes a method that includes identifying a repeating pattern of data having a number of cache lines that exceeds a storage capacity of a cache, and allocating storage in the cache to only a subset of the cache lines of the identified repeating pattern of data. In some examples, the method further includes bypassing remaining cache lines of the repeating pattern of data from storing in the cache. In some examples, allocating storage to only the subset includes selecting cache lines of the repeating pattern of data that exceed a maximum number of cache lines that the cache can store as remainder.

[0003] In some examples, identifying the repeating pattern includes determining that a reuse distance of cache lines of the repeating pattern of data is greater than the number of cache lines that can be stored in the cache, and limiting the subset to the number of cache lines that can be stored in the cache. In some examples, the method further includes iteratively selecting cache lines of the repeating pattern of data to evict from the cache, determining a hit rate for the subset of cache lines of the repeating pattern of data that are not evicted from the cache, and selecting cache lines of the repeating pattern of data to evict from the cache based on the hit rate.

[0004] In some examples, the method further includes partitioning the cache into two or more portions and allocating a first portion of the cache to a repeating pattern of data. Determining that a cache line reuse distance of the repeating pattern of data is greater than the number of cache lines that can be stored in the cache in some examples includes determining that a cache line reuse distance of the repeating pattern of data is greater than the number of cache lines that can be stored in the first portion of the cache. In some examples, the repeating pattern of data includes a texture.

[0005] Another exemplary method includes allocating entries of the cache to only a subset of the cache lines of the repeating pattern of data in response to determining that all cache lines of the repeating pattern of data would exceed the storage capacity of the cache. In some examples, the method further includes bypassing storing remainder of the cache lines of the repeating pattern of data in the cache. Allocating entries of the cache to only the subset in some examples includes selecting cache lines of the repeating pattern of data that exceed a maximum number of cache lines that the cache can store as remainder.

[0006] In some examples, the method further includes determining that a reuse distance of cache lines of the repeating pattern of data is greater than the number of cache lines that can be stored in the cache, and limiting the subset to the number of cache lines that can be stored in the cache. In some examples, the method further includes iteratively selecting cache lines of the repeating pattern of data to evict from the cache, determining a hit rate for cache lines of the repeating pattern of data that are not evicted from the cache, and selecting cache lines of the repeating pattern of data to evict from the cache based on the hit rate.

[0007] In some examples, the method further includes partitioning the cache into two or more portions and allocating a first portion of the cache to a repeating pattern of data. Determining that a cache line reuse distance of the repeating pattern of data is greater than the number of cache lines that can be stored in the cache includes determining that a cache line reuse distance of the repeating pattern of data is greater than the number of cache lines that can be stored in the first portion of the cache. In some examples, the repeating pattern of data includes a texture.

[0008] In another example, a device includes a parallel processing unit including a cache and a cache controller for identifying a repeating pattern of data having a number of cache lines that exceeds a storage capacity of the cache and for allocating storage in the cache to only a subset of the cache lines of the repeating pattern of data, the cache controller being, in some examples, for bypassing storing the remainder of the cache lines of the repeating pattern of data in the cache.

[0009] In some examples, the cache controller is configured to select cache lines of the repeating pattern of data that exceed a maximum number of cache lines that the cache can store as remainder. In some examples, the cache controller is configured to determine whether a reuse distance of the cache lines of the repeating pattern of data is greater than the number of cache lines that can be stored in the cache and limit the subset to the number of cache lines that can be stored in the cache. In one example, the cache controller is configured to iteratively select cache lines of the repeating pattern of data to evict from the cache, determine a hit rate for the cache lines of the repeating pattern of data that are not evicted from the cache, and select cache lines of the repeating pattern of data to evict from the cache based on the hit rate.

[0010] In another example, the cache controller is configured to partition the cache into two or more portions and allocate a first portion of the cache to the repeating pattern of data, and the cache controller is further configured to determine that a reuse distance of cache lines of the first repeating pattern of data is greater than the number of cache lines that can be stored in the cache by determining that a reuse distance of cache lines of the first repeating pattern of data is greater than the number of cache lines that can be stored in the cache.

[0011] The present disclosure may be better understood, and its numerous features and advantages made apparent to those skilled in the art by referencing the accompanying drawings, in which: The use of the same reference numbers in different drawings indicates similar or identical items. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a block diagram of a processing system configured to selectively cache a subset of repeating patterns of data, according to some embodiments. [Figure 2] FIG. 10 is a diagram of caching of repeating patterns of data without selecting a subset of the repeating patterns, according to some embodiments. [Figure 3] FIG. 1 is a diagram of caching repeating patterns of data, according to some embodiments. [Figure 4] FIG. 2 is a block diagram of a cache controller that allocates storage in a cache to only a subset of cache lines of repeating patterns of data, according to some embodiments. [Figure 5] FIG. 1 is a block diagram of a cache controller that partitions a cache and allocates storage in a portion of the cache to only a subset of cache lines of repeating patterns of data, according to some embodiments. [Figure 6] FIG. 1 illustrates selectively caching cache lines of repeating patterns of data according to some embodiments. [Figure 7] FIG. 1 illustrates selectively caching cache lines of repeating patterns of data according to some embodiments. [Figure 8] FIG. 1 is a flow diagram illustrating a method for selectively caching cache lines of repeating patterns of data, according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0013] Processor cores of a parallel processing unit often execute dispatches of work items that sequentially access repeating patterns of data, such as surfaces. To illustrate, a surface, such as a texture, attached to a series of dispatches of work items is read in successive rendering or denoising passes. For example, many elements of a video game scene are rendered in successive frames as the camera view position or angle changes. Successive passes require the same texture data to map to pixels. In some cases, the size of the texture exceeds the storage capacity of the parallel processing unit's lowest level cache (L1 cache).

[0014] Traditionally, a cache controller for an L1 cache employs an LRU cache replacement policy, fetching data requested by a processor core that is not currently stored in the L1 cache (i.e., a cache miss) from a higher-level cache in the cache hierarchy. To create room for the requested cache line, the least recently used cache line in the L1 cache is evicted. However, if the reuse distance (i.e., the number of cache lines in each repetition of the pattern) of a repeating pattern of data exceeds the storage capacity of the L1 cache, the cache line will already have been evicted from the cache by the time a new request for a cache line in the repeating pattern of data is received by the cache controller, resulting in a low (or zero) hit rate. Retrieving data from more distant levels of the memory hierarchy so that the data can be accessed by subsequent dispatches consumes resources and increases latency.

[0015] 1-8 illustrate techniques for a processing system for selectively allocating storage in a local cache of a parallel processing unit to cache lines of a repeating pattern of data when the repeating pattern is of a size that exceeds the storage capacity of the cache. The processing system identifies a repeating pattern of data having cache lines with a reuse distance that exceeds the storage capacity of the cache. The cache controller allocates storage to only a subset (i.e., some, but not all) of the cache lines of the repeating pattern of data in the cache and excludes the remainder of the cache lines of the repeating pattern of data from the cache. For example, if the maximum number of cache lines a cache can store at one time is X cache lines and the processing system identifies a repeating pattern of data that is X+Y cache lines, in some embodiments, the cache controller allocates storage to only the first X cache lines of the repeating pattern of data and excludes the remaining Y cache lines from being stored in the cache. In response to a request from a processor core for the remaining Y cache lines, the cache controller provides the remaining Y cache lines directly to the processor core from a higher level of the cache hierarchy, bypassing storing the Y cache lines in the cache. By limiting the cache to store only the first X cache lines of a repeating pattern of data, the cache controller increases the hit rate in the cache for the first X cache lines, thus reducing latency and improving the user experience.

[0016] In some embodiments, the processing system probabilistically selects a subset of cache lines from the repeating pattern of data to store in the cache in an iterative process. The processing system measures a hit rate for each of the selected subset of cache lines and selects cache lines of the repeating pattern of data to remove from the cache based on the hit rate. In some embodiments, the cache controller partitions the cache into two or more portions, allocating one portion of the cache to store a subset of cache lines of a first repeating pattern of data and another portion of the cache to store a subset of cache lines of a second repeating pattern of data. For the partitioned cache, the cache controller compares the reuse distance of the cache lines of the first repeating pattern of data to the storage capacity of the portion of the cache to which the first repeating pattern of data is allocated. Based on the reuse distance and the storage capacity of the portion of the cache, the cache controller selects the subset of cache lines of the first repeating pattern of data to store in the cache and removes the remaining cache lines of the first repeating pattern of data from the cache.

[0017] 1 illustrates a processing system 100 configured to selectively allocate storage in a local cache of a parallel processing unit to cache lines of repeating patterns of data that exceed the storage capacity of the cache, according to some embodiments. Processing system 100 includes a parallel processing unit 110, such as a graphics processing unit (GPU) for generating visual images intended for output to a display 175, according to some embodiments. A parallel processor is a processor that can execute a single instruction on multiple data or threads in parallel.

[0018] Examples of parallel processors include processors such as graphics processing units (GPUs), massively parallel processors, single instruction, multiple data (SIMD) architecture processors, and single instruction, multiple threads (SIMT) architecture processors for performing graphics, machine intelligence, or computational operations. In some embodiments, the parallel processor is a separate device included as part of a computer. In other embodiments, such as an advanced processor unit, the parallel processor is included in a single device along with a host processor, such as a central processor unit (CPU). The following description uses a graphics processing unit (GPU) for illustrative purposes, although the embodiments described below are applicable to other types of parallel processors.

[0019] Processing system 100 includes system memory 150 (referred to herein as memory 150). Some embodiments of memory 150 are implemented as dynamic random access memory (DRAM). However, memory 150 may also be implemented using other types of memory, including static random access memory (SRAM), non-volatile RAM, etc. In the illustrated embodiment, parallel processing unit 110 communicates with memory 150 via bus 160. However, some embodiments of parallel processing unit 110 communicate with memory 150 via a direct connection or through other buses, bridges, switches, routers, etc. Parallel processing unit 110 executes instructions stored in memory 150, and parallel processing unit 110 stores information, such as results of executed instructions, in memory 150. For example, memory 150 may store copies of instructions from an application 155 executed by parallel processing unit 110. Some embodiments of parallel processing unit 110 include multiple processor cores (called compute units) 115 that independently execute instructions simultaneously or in parallel.

[0020] Processing system 100 is generally configured to execute instruction sets (e.g., computer programs), such as applications 155, to perform designated tasks for the electronic device. Examples of such tasks include controlling aspects of the electronic device's operation, displaying information to a user to provide a particular user experience, communicating with other electronic devices, etc. Accordingly, in different embodiments, processing system 100 is employed in any of numerous types of electronic devices, such as a desktop computer, a laptop computer, a server, a game console, a tablet, a smartphone, etc. Components of processing system 100 are implemented as hardware, firmware, software, or any combination thereof. It should be understood that processing system 100 may include more or fewer components than those shown in FIG. 1 . For example, processing system 100 may further include one or more input interfaces, non-volatile storage, one or more output interfaces, a network interface, and one or more displays or display interfaces.

[0021] Processing system 100 includes a central processing unit (CPU) 105 for executing instructions. Some embodiments of CPU 105 include multiple processor cores (not shown for clarity) that independently execute instructions simultaneously or in parallel. CPU 105 is also connected to bus 160 and thus communicates with parallel processing units 110 and memory 150 via bus 160. CPU 105 executes instructions, such as program code of application 155 stored in memory 150, and CPU 105 stores information, such as results of executed instructions, in memory 150. CPU 105 can also initiate graphics operations by issuing draw calls to parallel processing units 110. A draw call is a command generated by CPU 105 and sent to parallel processing units 110 to instruct the parallel processing units 110 to render an object (or portion of an object) within a frame. Some embodiments of a draw call include information defining textures, states, shaders, rendering objects, buffers, etc., used by parallel processing units 110 to render the object or portion thereof. Parallel processing unit 110 renders the object to generate pixel values ​​that are provided to display 175, which uses the pixel values ​​to display an image representing the rendered object.

[0022] In some embodiments, each frame to be rendered is processed by the graphics pipeline of parallel processing unit 110 in multiple passes. For example, during a first pass over the scene's geometry, only the attributes necessary to calculate per-pixel lighting are written to the G-buffer. During a second pass, the graphics pipeline outputs only diffuse and specular lighting data. On the frame's third pass through the graphics pipeline, the graphics pipeline reads backlight data and outputs the final per-pixel shading. Thus, in multi-pass rendering, a frame's scene and associated objects are rendered multiple times. Each time an object is drawn, the graphics pipeline calculates additional aspects of the object's appearance and combines the additional aspects with previous results. Each time a frame or a frame's objects is rendered by the graphics pipeline is referred to as a rendering pass.

[0023] Input / output (I / O) engine 170 handles input or output operations associated with display 175 and other elements of processing system 100, such as a keyboard, mouse, printer, external disk, etc. I / O engine 170 is coupled to bus 160 such that I / O engine 170 communicates with parallel processing unit 110, memory 150, or CPU 105. In the illustrated embodiment, I / O engine 170 is configured to read information stored on an external storage medium 180, such as a compact disc (CD), digital video disc (DVD), or the like. External storage medium 180 stores information representing program code used to implement applications, such as video games. Program code on external storage medium 180 can be written to memory 150 to form a copy of instructions to be executed by parallel processing unit 110 or CPU 105.

[0024] In some embodiments, parallel processing unit 110 implements a graphics pipeline (not shown in FIG. 1 for clarity) that includes multiple stages configured to process different primitives simultaneously in response to draw calls. The stages of the graphics pipeline within parallel processing unit 110 can simultaneously process different primitives generated by an application, such as a video game. When geometry is presented to the graphics pipeline, hardware state settings are selected to define the state of the graphics pipeline. Examples of state include rasterizer state, blend state, depth stencil state, primitive topology type of the presented geometry, and shaders (e.g., vertex shader, domain shader, geometry shader, hull shader, pixel shader, etc.) used to render the scene. Shaders implemented in the graphics pipeline state are represented by corresponding bytecode. In some cases, information representing the graphics pipeline state is hashed or compressed to provide a more efficient representation of the graphics pipeline state.

[0025] Driver 165 is a computer program that allows higher-level graphics computing programs, such as from application 155, to interact with parallel processing unit 110. For example, driver 165 converts standard code received from application 155 into a native format command stream understood by parallel processing unit 110. Driver 165 allows input from application 155 to dictate settings of parallel processing unit 110. Such settings include selection of rendering mode, anti-aliasing control, texture filter control, batch binning control, and deferred pixel shading control.

[0026] To execute sets of commands received from the CPU, parallel processing unit 110 includes multiple compute units 115 and L1 cache 120. The multiple compute units 115 together perform shading operations for dispatches of work items (not shown). In different embodiments, the compute units 115 perform geometry operations, texture operations, tessellation operations, vertex operations, mesh operations, primitive operations, ray tracing operations, compute operations, etc., or any combination thereof, based on commands received from a command processor (not shown). In some embodiments, to perform these operations, each of the compute units 115 includes one or more SIMD elements configured to perform the specified operations using the work items of the received dispatches.

[0027] L1 cache 120 stores data for multiple compute units 115. Thus, in the course of performing shader operations, multiple compute units 115 store and retrieve data from L1 cache 120, where the data stored and retrieved is based on the particular work item being processed. For example, in some embodiments, each work item in a dispatch corresponds to an individual pixel of an image, and L1 cache 120 stores data (e.g., texture values) for each individual pixel or a subset of the individual pixels included in the dispatch. In some embodiments, parallel processing unit 110 is associated with a memory hierarchy having multiple cache levels and system memory 150, and L1 cache 120 represents the lowest of the multiple cache levels.

[0028] To increase the hit rate in L1 cache 120 for repeating patterns of data that exceed the storage capacity of L1 cache 120, parallel processing unit 110 includes a cache controller 130 configured to allocate storage in L1 cache 120 to only a subset 140 of cache lines of the repeating pattern of data and to exclude the remainder of the cache lines of the repeating pattern of data from caching. In response to a request from compute unit 115 for the remainder of the cache lines of the repeating pattern of data, cache controller 130 provides the remaining cache lines directly to compute unit(s) 115 from a higher level of the cache hierarchy (not shown), bypassing storage of the remaining cache lines in L1 cache 120. By restricting L1 cache 120 to store only cache lines of the repeating pattern of data that simultaneously fit within the storage capacity of L1 cache 120, cache controller 130 increases the hit rate in L1 cache 120 for the selected subset 140 of cache lines.

[0029] The cache controller 130 includes a pattern recognition unit 135 configured to identify a repeating pattern of data that exceeds the storage capacity of the L1 cache 120. The pattern recognition unit 135 measures the reuse distance of cache lines in the repeating pattern of data (i.e., the number of cache lines in each repetition of the pattern) and compares the reuse distance DR with the total number CL1 of cache lines that the L1 cache 120 can store (i.e., the storage capacity of the L1 cache 120). If the reuse distance DR exceeds the storage capacity CL1 of the L1 cache 120, the cache controller 130 limits the number of cache lines of the repeating pattern of data allocated to storage space in the L1 cache 120 to a subset 140 of cache lines that fit within the storage capacity CL1 of the L1 cache 120. The cache controller 130 excludes the remaining (DR-CL1) cache lines of the repeating pattern of data from the L1 cache 120. In some embodiments, in response to L1 cache 120 exhibiting a relatively low hit rate, pattern recognition unit 135, in an iterative process, probabilistically selects a subset of cache lines from the repeating pattern of data to store in L1 cache 120. Cache controller 130 measures the hit rate for each of the selected subset of cache lines and selects cache lines from the repeating pattern of data to evict from the cache based on the hit rate.

[0030] In some embodiments, each draw call is associated with an identifier (not shown), and the request stream for a succession of dispatches of work items includes the draw call identifier and the surface on which the draw call will operate. Pattern recognition unit 135 tracks the number of requests for draw call identifier / surface combinations that have been evicted from L1 cache 120. Pattern recognition unit 135 enumerates draw call identifier / surface combinations according to how frequently they are evicted from L1 cache 120. In some embodiments, cache controller 130 simulates hit rates for a selected subset of cache lines 140 based on the inclusion or exclusion of draw call identifier / surface combinations from L1 cache 120 to determine which draw call identifier / surface combinations to evict from L1 cache 120 to increase hit rates.

[0031] In some embodiments, cache controller 130 partitions L1 cache 120 into two or more portions (not shown) and allocates one portion of L1 cache 120 to store a subset of cache lines of a first repeating pattern of data and another portion of L1 cache 120 to store a subset of cache lines of a second repeating pattern of data. For the partitioned L1 cache 120, cache controller 130 compares the reuse distance DR of the cache lines of the first repeating pattern of data to the storage capacity of the portion of the cache to which the first repeating pattern of data is allocated. Based on the reuse distance DR and the storage capacity of the portion of the cache, cache controller 130 selects the subset of cache lines of the first repeating pattern of data to store in the cache and excludes the remaining cache lines of the first repeating pattern of data from the cache.

[0032] 2 is an illustration 200 of caching repeating patterns of data without selecting a subset of cache lines based on reuse distance, according to some embodiments. In the illustrated example, L1 cache 120 is sized to hold up to three cache lines at a time. After L1 cache 120 is flushed, at time T1, cache controller 130 receives a request for cache line-1 202 from compute unit 115. Because L1 cache 120 does not contain valid data, the request misses in L1 cache 120, and cache controller 130 copies cache line-1 202 from L2 cache 220, a higher level cache in the cache hierarchy, into L1 cache 120 and provides cache line-1 202 to compute unit 115. At a subsequent time T2, cache controller 130 receives a request for cache line-2 204 from compute unit 115. Because L1 cache 120 does not contain cacheline-2 204, the request misses in L1 cache 120, and cache controller 130 copies cacheline-2 204 from L2 cache 220 to L1 cache 120 and provides cacheline-2 204 to compute unit 115. At a subsequent time T3, the cache controller receives a request for cacheline-3 206 from compute unit 115. Because L1 cache 120 does not contain cacheline-3 206, the request misses in L1 cache 120, and cache controller 130 copies cacheline-3 206 from L2 cache 220 to L1 cache 120 and provides cacheline-3 206 to compute unit 115.

[0033] At a subsequent time T4, cache controller 130 receives a request for cacheline-4 208 from compute unit 115. The request misses in L1 cache 120 because L1 cache 120 does not contain cacheline-4 208. To create room in L1 cache 120 for cacheline-4 208, the cache controller evicts cacheline-1 202 because it is the least recently used cache line. Cache controller 130 copies cacheline-4 208 from L2 cache 220 to L1 cache 120 and provides cacheline-4 208 to compute unit 115.

[0034] At a subsequent time T5, cache controller 130 receives a request for cache line-5 210 from compute unit 115. Because L1 cache 120 does not contain cache line-5 210, the request misses in L1 cache 120. To create room in L1 cache 120 for cache line-5 208, the cache controller evicts cache line-2 204 because it is the least recently used cache line. Cache controller 130 copies cache line-5 210 from L2 cache 220 to L1 cache 120 and provides cache line-5 210 to compute unit 115. Therefore, for the first five cycles T1 to T5 of access requests to cache lines 202, 204, 206, 208, and 210, the cache hit rate is 0%.

[0035] 3 is an illustration 300 of the continuation of caching the repeating pattern of data of FIG. 2. At a subsequent time T6, cache controller 130 receives a request for cacheline-1 202 from compute unit 115. The request misses in L1 cache 120 because L1 cache 120 does not contain cacheline-1 202 (because cacheline-1 202 was previously evicted at time T4). To create room in L1 cache 120 for cacheline-1 202, cache controller 130 evicts cacheline-3 206 because it is the least recently used cache line. Cache controller 130 copies cacheline-1 202 from L2 cache 220 to L1 cache 120 and provides cacheline-1 202 to compute unit 115.

[0036] At a subsequent time T7, cache controller 130 receives a request for cacheline-2 204 from compute unit 115. The request misses in L1 cache 120 because L1 cache 120 does not contain cacheline-2 204 (because cacheline-2 202 was previously evicted at time T5). To create room in L1 cache 120 for cacheline-2 204, cache controller 130 evicts cacheline-4 208 because it is the least recently used cache line. Cache controller 130 copies cacheline-2 204 from L2 cache 220 to L1 cache 120 and provides cacheline-2 204 to compute unit 115.

[0037] At a subsequent time T8, cache controller 130 receives a request for cache line-3 206 from compute unit 115. The request misses in L1 cache 120 because L1 cache 120 does not contain cache line-3 206 (because cache line-3 206 was previously evicted at time T6). To create room in L1 cache 120 for cache line-3 206, cache controller 130 evicts cache line-5 210 because it is the least recently used cache line. Cache controller 130 copies cache line-3 206 from L2 cache 220 to L1 cache 120 and provides cache line-3 206 to compute unit 115.

[0038] At a subsequent time T9, cache controller 130 receives a request for cacheline-4 208 from compute unit 115. The request misses in L1 cache 120 because L1 cache 120 does not contain cacheline-4 208 (because cacheline-4 208 was previously evicted at time T7). To create room in L1 cache 120 for cacheline-4 208, the cache controller evicts cacheline-1 202 because it is the least recently used cache line. Cache controller 130 copies cacheline-4 208 from L2 cache 220 to L1 cache 120 and provides cacheline-4 208 to compute unit 115.

[0039] At a subsequent time T10, cache controller 130 receives a request for cache line-5 210 from compute unit 115. The request misses in L1 cache 120 because L1 cache 120 does not contain cache line-4 208 (because cache line-5 210 was previously evicted at time T8). To create room in L1 cache 120 for cache line-5 210, the cache controller evicts cache line-2 204 because it is the least recently used cache line. Cache controller 130 copies cache line-5 210 from L2 cache 220 to L1 cache 120 and provides cache line-5 210 to compute unit 115. Thus, for a five cache line pattern of repeating data in cache lines 202, 204, 206, 208, and 210, caching using the LRU replacement policy results in a 0% hit rate for L1 cache 120 for cycles T6 through T10, no improvement over the 0% hit rate for the cold start scenario shown in FIG. 2.

[0040] 4 is a block diagram 400 of cache controller 130 of FIG. 1 that allocates storage in L1 cache 120 to only a subset 140 of cache lines of a repeating pattern 420 of data, according to some embodiments. In the illustrated example, L1 cache 120 is sized to hold up to three cache lines at a time. The repeating pattern 420 of data includes five cache lines (1, 2, 3, 4, 5, 1, 2, 3, 4, 5, 1, 2, 3, 4, 5, ...), thus exceeding the storage capacity of L1 cache 120. Cache controller 130 employs a cache replacement policy 405 that evicts the least recently used cache line if L1 cache 120 is full when a cache line is fetched from a higher level cache.

[0041] The pattern recognition unit 135 identifies a repeating pattern of data 420 when data is fetched from the GPU memory 410 and determines that the cache lines of the repeating pattern of data 420 have a reuse distance that exceeds the storage capacity of the L1 cache 120. In response to the pattern recognition unit 135 determining that the cache lines of the repeating pattern of data 420 have a reuse distance that exceeds the storage capacity of the L1 cache 120, the cache controller 130 allocates storage for only the subset of cache lines 140 (e.g., cache lines 1, 2, and 3) that can be stored in the L1 cache 120 at one time and excludes the remainder 415 of the cache lines of the repeating pattern of data 420 (cache lines 4 and 5) from the L1 cache 120. Thus, when the compute unit 115 requests each of cache lines 1, 2, and 3 of the subset of cache lines 140, the cache controller allocates entries for cache lines 1, 2, and 3 and stores them in the L1 cache 120. However, when the compute unit 115 requests the remainder of the cache line 415 (cache lines 4 and 5), the cache controller provides the remainder of the cache line 415 directly from the L2 cache 220 to the compute unit 115, bypassing storing the remainder of the cache line 415 in the L1 cache 120.

[0042] 5 is a block diagram 500 of a cache controller 130 that partitions L1 cache 120 and allocates storage in a portion of the cache to only a subset of cache lines of a repeating pattern of data, according to some embodiments. At times, parallel processing unit 110 processes draw calls that access two or more textures in parallel. In the illustrated example, cache controller 130 fetches two textures from GPU memory 410 for overlapping access: a first repeating pattern of data 520 and a second repeating pattern of data 530. To increase the hit rate for first repeating pattern of data 520 and second repeating pattern of data 530, cache controller 130 partitions L1 cache 120 into a first portion 540 and a second portion 545.

[0043] In response to pattern recognition unit 135 identifying a first repeating pattern of data 520 and determining that a cache line reuse distance 550 of first repeating pattern of data 520 exceeds a storage capacity 555 of a first portion 540 of the L1 cache, cache controller 130 allocates storage in first portion 540 for storing a subset 525 of cache lines of first repeating pattern of data 520. Similarly, in response to pattern recognition unit 135 identifying a second repeating pattern of data 530 and determining that a cache line reuse distance 560 of second repeating pattern of data 530 exceeds a storage capacity 565 of a second portion 545 of the L1 cache, cache controller 130 allocates storage in second portion 545 for storing a subset 535 of cache lines of second repeating pattern of data 530.

[0044] FIG. 6 is an illustration 600 of selectively caching cache lines of a repeating pattern of data, according to some embodiments. Similar to illustration 200 of FIG. 2, the repeating pattern of data includes five cache lines, but L1 cache 120 has a storage capacity of three cache lines. However, unlike illustration 200 of FIG. 2, cache controller 130 performs selective caching for the repeating pattern of data in the illustrated example. In response to pattern recognition unit 135 determining that the cache lines of the repeating pattern of data have a reuse distance that exceeds the storage capacity of L1 cache 120, the cache controller allocates space in L1 cache 120 to only a subset of the cache lines of the repeating pattern of data (cache lines 202, 204, 206) and excludes the remainder of the cache lines of the repeating pattern of data from L1 cache 120.

[0045] After L1 cache 120 is flushed, at time T1, cache controller 130 receives a request for cacheline-1 202 from compute unit 115. Because L1 cache 120 does not contain valid data, the request misses in L1 cache 120. Because cacheline-1 202 is included in the subset of cache lines for which cache controller 130 has allocated storage in L1 cache 120, cache controller 130 copies cacheline-1 202 from L2 cache 220 to L1 cache 120 and provides cacheline-1 202 to compute unit 115.

[0046] At a subsequent time T2, cache controller 130 receives a request for cacheline-2 204 from compute unit 115. Because L1 cache 120 does not include cacheline-2 204, the request misses in L1 cache 120. Because cache controller 130 has included cacheline-2 204 in the subset of cachelines for which storage has been allocated in L1 cache 120, cache controller 130 copies cacheline-2 204 from L2 cache 220 to L1 cache 120 and provides cacheline-2 204 to compute unit 115.

[0047] At a subsequent time T3, the cache controller receives a request for cacheline-3 206 from the compute unit 115. Because the L1 cache 120 does not include cacheline-3 206, the request misses in the L1 cache 120. Because the cache controller 130 has included cacheline-3 206 in the subset of cachelines for which storage has been allocated in the L1 cache 120, the cache controller 130 copies cacheline-3 206 from the L2 cache 220 to the L1 cache 120 and provides cacheline-3 206 to the compute unit 115.

[0048] At a subsequent time T4, cache controller 130 receives a request for cacheline-4 208 from compute unit 115. Because L1 cache 120 does not include cacheline-4 208, the request misses in L1 cache 120. Because cache controller 130 does not include cacheline-4 208 in the subset of cache lines of the repeating pattern of data that has storage allocated in L1 cache 120, the cache controller provides cacheline-4 208 directly from L2 cache 220 to compute unit 115 and excludes cacheline-4 208 from L1 cache 120.

[0049] At a subsequent time T5, cache controller 130 receives a request for cache line-5 210 from compute unit 115. Because L1 cache 120 does not include cache line-5 210, the request misses in L1 cache 120. Because cache controller 130 does not include cache line-5 210 in the subset of cache lines for the repeating pattern of data that has storage allocated in L1 cache 120, the cache controller provides cache line-5 210 directly from L2 cache 220 to compute unit 115 and excludes cache line-5 210 from L1 cache 120. Thus, similar to illustration 200 of FIG. 2, the cache hit rate is 0% for the first five cycles T1-T5 of access requests to cache lines 202, 204, 206, 208, and 210.

[0050] Figure 7 is an illustration 700 of the continuation of selective caching of the repeating pattern of data of Figure 6. At a subsequent time T6, cache controller 130 receives a request for cacheline-1 202 from compute unit 115. Because L1 cache 120 includes cacheline-1 202, the request results in a hit in L1 cache 120, and cache controller 130 provides cacheline-1 202 to compute unit 115. At a subsequent time T7, cache controller 130 receives a request for cacheline-2 204 from compute unit 115. Because L1 cache 120 includes cacheline-2 204, the request results in a hit in L1 cache 120, and cache controller 130 provides cacheline-2 204 to compute unit 115. At a subsequent time T8, cache controller 130 receives a request for cacheline-3 206 from compute unit 115. Because L1 cache 120 contains cache line-3 206 , the request results in a hit in L1 cache 120 and cache controller 130 provides cache line-3 206 to compute unit 115 .

[0051] At a subsequent time T9, cache controller 130 receives a request for cache line-4 208 from compute unit 115. Because cache controller 130 has excluded cache line-4 208 from L1 cache 120, L1 cache 120 does not contain cache line-4 208. Therefore, the request misses in L1 cache 120. Cache controller 130 fetches cache line-4 208 from L2 cache 220 and provides cache line-4 208 directly to compute unit 115, bypassing storing cache line-4 208 in L1 cache 120 because cache line-4 208 is not included in the subset of cache lines for which space is allocated in L1 cache 120.

[0052] At a subsequent time T10, cache controller 130 receives a request for cache line-5 210 from compute unit 115. Because cache controller 130 has excluded cache line-5 210 from L1 cache 120, L1 cache 120 does not contain cache line-5 210. Thus, the request misses in L1 cache 120. Cache controller 130 fetches cache line-5 210 from L2 cache 220 and provides cache line-5 210 directly to compute unit 115, bypassing storing cache line-5 210 in L1 cache 120 because cache line-5 210 is not included in the subset of cache lines for which space is allocated in L1 cache 120. Thus, for a repeating five cache line pattern of data in cache lines 202, 204, 206, 208, and 210, selective caching using an LRU replacement policy results in a 60% hit rate for L1 cache 120 for cycles T6-T10, which is a significant increase over the 0% hit rate for the conventional caching scenario shown in Figure 3. Furthermore, additional cycles of the same repeating pattern of data can be expected to result in a similar increased hit rate for L1 cache 120 using selective caching.

[0053] 8 is a flow diagram illustrating a method 800 for selectively caching cache lines of repeating patterns of data, according to some embodiments. In some embodiments, method 800 is performed by a processing system, such as processing system 100 shown in FIG. 1. At block 802, pattern recognition unit 135 identifies repeating patterns of data 420. At block 804, pattern recognition unit 135 compares the reuse distance of the cache lines of repeating patterns of data 420 to the maximum number of cache lines that L1 cache 120 can store at one time and determines whether the reuse distance exceeds the storage capacity of L1 cache 120.

[0054] If, at block 804, the pattern recognition unit 135 determines that the reuse distance of the cache lines of the repeating pattern of data 420 does not exceed the storage capacity of the L1 cache 120, method flow continues to block 806. At block 806, the cache controller 130 caches all cache lines of the repeating pattern of data in the L1 cache 120.

[0055] If, at block 804, the pattern recognition unit 135 determines that the reuse distance of the cache lines of the repeating pattern of data 420 exceeds the storage capacity of the L1 cache 120, method flow continues to block 808. At block 808, the cache controller 130 allocates storage in the L1 cache 120 to only a subset 140 of the cache lines of the repeating pattern of data 420 and evicts the remainder 415 of the cache lines of the repeating pattern of data 420 from the L1 cache 120. In some embodiments, the pattern recognition unit 135 probabilistically selects the subset of cache lines from the repeating pattern of data to store in the L1 cache 120 in an iterative process. The cache controller 130 measures a hit rate for each of the selected subset of cache lines and selects the subset 140 of the cache lines of the repeating pattern of data 420 to evict from the L1 cache 120 based on the hit rate.

[0056] In some embodiments, cache controller 130 partitions L1 cache 120 into a first portion 540 and a second portion 545. In response to pattern recognition unit 135 identifying first repeating pattern of data 520 and determining that the reuse distance of cache lines of first repeating pattern of data 520 exceeds the storage capacity of first portion 540 of L1 cache, cache controller 130 allocates storage in first portion 540 for storing a subset 525 of cache lines of first repeating pattern of data 520.

[0057] In some embodiments, the above-described apparatus and techniques are implemented in a system including one or more integrated circuit (IC) devices (also referred to as integrated circuit packages or microchips), such as the processing systems described above with reference to FIGS. 1-8. Electronic design automation (EDA) and computer-aided design (CAD) software tools can be used in the design and manufacture of these IC devices. These design tools are typically represented as one or more software programs. The one or more software programs include code executable by a computer system for operating the computer system to operate on code representing the circuits of one or more IC devices to perform at least a portion of a process for designing or adapting a manufacturing system for producing the circuits. This code may include instructions, data, or a combination of instructions and data. The software instructions representing the design or manufacturing tools are typically stored in a computer-readable storage medium accessible to the computing system. Similarly, code representing one or more stages of the design or manufacture of the IC devices is stored in and accessed from the same or a different computer-readable storage medium.

[0058] A computer-readable storage medium includes any non-transitory storage medium or combination of non-transitory storage media that can be accessed by a computer system during use to provide instructions and / or data to the computer system. Such storage media may include, but are not limited to, optical media (e.g., compact discs (CDs), digital versatile discs (DVDs), Blu-ray discs), magnetic media (e.g., floppy disks, magnetic tape, magnetic hard drives), volatile memory (e.g., random access memory (RAM) or cache), non-volatile memory (e.g., read-only memory (ROM) or flash memory), or micro-electromechanical systems (MEMS)-based storage media. The computer-readable storage medium (e.g., system RAM or ROM) may be internal to the computing system, the computer-readable storage medium (e.g., a magnetic hard drive) may be permanently attached to the computing system, the computer-readable storage medium (e.g., an optical disk or Universal Serial Bus (USB)-based flash memory) may be removably attached to the computing system, or the computer-readable storage medium (e.g., network-accessible storage (NAS)) may be coupled to the computer system via a wired or wireless network.

[0059] In some embodiments, certain aspects of the techniques described above are implemented by one or more processors of a processing system executing software. The software includes one or more sets of executable instructions stored or otherwise tangibly embodied in a non-transitory computer-readable storage medium. The software may include instructions and specific data that, when executed by one or more processors, operate the one or more processors to perform one or more aspects of the techniques described above. The non-transitory computer-readable storage medium may include, for example, a magnetic or optical disk storage device, a solid-state storage device such as flash memory, a cache, a random access memory (RAM), or other non-volatile memory device(s). The executable instructions stored on the non-transitory computer-readable storage medium may be implemented as source code, assembly language code, object code, or other form of instructions that can be interpreted or otherwise executed by one or more processors.

[0060] In addition to the above, it should be noted that not all activities or elements described in the summary description are required, that some of the particular activities or devices may not be required, that one or more additional activities may be performed, and that one or more additional elements may be included. Furthermore, the order in which the activities are listed is not necessarily the order in which they are performed. Also, the concepts have been described with reference to specific embodiments. However, those skilled in the art will recognize that various modifications and variations can be made without departing from the scope of the invention as set forth in the claims. Accordingly, the specification and drawings should be regarded in an illustrative rather than a restrictive sense, and all such modifications are intended to be included within the scope of the present invention.

[0061] Benefits, other advantages, and solutions to problems have been described above with regard to specific embodiments. However, the benefits, advantages, solutions to problems, and features from which any benefit, advantage, or solution may arise or be manifested are not construed as critical, essential, or essential features of any or all claims. Moreover, the specific embodiments described above are illustrative only, since the disclosed invention may be modified and practiced in different, but similar manners apparent to those skilled in the art having the benefit of the teachings herein. No limitations are intended to the details of construction or design herein shown, other than as described in the appended claims. It is therefore apparent that the specific embodiments described above may be altered or modified, and that all such variations are considered within the scope of the disclosed invention. Accordingly, the protection sought herein is set forth in the appended claims.

Claims

1. in response to a single instance of a repeating pattern of data spanning a number of cache lines that exceeds a storage capacity of the cache in which the repeating pattern of data is stored, allocating storage in the cache to only a subset of the cache lines for the single instance; method.

2. bypassing storing remaining cache lines of the single instance in the cache; 10. The method of claim 1.

3. Allocating storage to only the subset includes: selecting cache lines of the repeating pattern of the data that exceed a maximum number of cache lines that the cache can store as the remaining cache lines; The method of claim 2.

4. determining that a reuse distance of the single instance of a cache line is greater than the number of cache lines that can be stored in the cache; and limiting the subset to the number of cache lines that can be stored in the cache.

10. The method of claim 1.

5. determining a hit rate for a subset of cache lines of the repeating pattern of the data that is not excluded from the cache; selecting cache lines of the repeating pattern of the data to remove from the cache based on the hit rate; iteratively selecting cache lines of the repeating pattern of data to evict from the cache by The method of claim 4.

6. partitioning the cache into two or more portions; allocating a first portion of the cache to a repeating pattern of the data; determining that the reuse distance of the single-instance cache line is greater than the number of cache lines that can be stored in the cache includes determining that the reuse distance of the single-instance cache line is greater than the number of cache lines that can be stored in the first portion of the cache; The method of claim 4.

7. the repeating pattern of data comprises a texture; 10. The method of claim 1.

8. A device, a parallel processing unit; The parallel processing unit a cache in which repeating patterns of data are stored; a cache controller; The cache controller In response to a single instance of a repeating pattern of data having a number of cache lines that exceeds a storage capacity of the cache, allocating storage in the cache to only a subset of the cache lines of the single instance. device.

9. The cache controller bypassing the remaining cache lines of the repeating pattern of data from being stored in the cache; The device of claim 8.

10. The cache controller selecting, as the remaining cache lines, cache lines of the repeating pattern of the data that exceed a maximum number of cache lines that the cache can store; The device of claim 9.

11. The cache controller determining whether a cache line reuse distance of the single instance is greater than the number of cache lines that can be stored in the cache; limiting the subset to the number of cache lines that can be stored in the cache; The device of claim 8.

12. The cache controller determining a hit rate for a subset of cache lines of the repeating pattern of the data that is not excluded from the cache; selecting cache lines of the repeating pattern of the data to remove from the cache based on the hit rate; configured to: The device of claim 11.

13. The cache controller partitioning the cache into two or more portions; allocating a first portion of the cache to the repeating pattern of data; determining that a reuse distance of a cache line of the single instance is greater than a number of cache lines that can be stored in the cache by determining that a reuse distance of a cache line of the single instance is greater than a number of cache lines that can be stored in the cache; configured to: The device of claim 11.

14. the repeating pattern of data comprises a texture; The device of claim 8.

Images