Stochastic optimization of surface caching capabilities in parallel processing units

Abstract

The processing system [100] selectively allocates storage to the local cache [120] of the parallel processing unit [110] for cache lines of a data repetition pattern [420] that exceed the storage capacity of the cache. The processing system identifies a data repetition pattern that has cache lines with a reuse distance that exceeds the storage capacity of the cache. The cache controller [130] allocates storage to only a subset of cache lines [140] of the data repetition pattern in the cache and excludes the remaining cache lines [415] of the data repetition pattern from the cache. By restricting the cache to store only a subset of cache lines of the data repetition pattern, the cache controller increases the cache hit rate for the subset of cache lines.

Description

Background Technology

[0001] Processing systems, including parallel processing units such as graphics processing units (GPUs), implement a cache hierarchy (or multi-level cache) that uses a hierarchy of caches of varying speeds to store frequently accessed data. Data that is requested more frequently is typically cached in a relatively high-speed cache (such as the L1 cache) that is physically (or logically) located close to the processor core or computing unit. Higher-level caches (such as the L2 cache, L3 cache, etc.) store data that is requested less frequently. The Last Level Cache (LLC) is the highest-level (and lowest access rate) cache, and the LLC reads data directly from system memory and writes data directly to system memory. The cache differs from system memory because it implements a cache replacement policy that replaces data in a cache line in response to new data that needs to be written to that cache line. For example, the Lowest Used-Router (LRU) policy replaces data in a cache line that has not been accessed for the longest time interval by removing data from the LRU cache line and writing new data to the LRU cache line.

[0002] One example includes a method comprising the steps of identifying a repeating pattern of data having a number of cache lines exceeding the storage capacity of the cache, and allocating storage to only a subset of the cache lines of the identified repeating pattern of data in the cache. In some examples, the method further comprises the step of bypassing the storage of the remaining cache lines of the repeating pattern of data in the cache. In some examples, the step of allocating storage to only a subset comprises the step of selecting the cache lines of the repeating pattern of data that exceed the maximum number of cache lines that the cache can store as a remainder.

[0003] In some examples, the step of identifying a repeating pattern includes determining that the reuse distance of a cache line of a repeating pattern of data exceeds the number of cache lines that can be stored in the cache, and limiting a subset to the number of cache lines that can be stored in the cache. In some examples, the method further includes the steps of repeatedly selecting a cache line of a repeating pattern of data to be excluded from the cache, determining a hit rate for a subset of cache lines of a repeating pattern of data not excluded from the cache, and selecting a cache line of a repeating pattern of data to be excluded from the cache based on the hit rate.

[0004] In some examples, the method further includes the steps of dividing the cache into two or more parts and assigning the first part of the cache to a repeating pattern of data. The step of determining that the reuse distance of the cache lines of the repeating pattern of data exceeds the number of cache lines that can be stored in the cache includes the step of determining that the reuse distance of the cache lines of the repeating pattern of data exceeds the number of cache lines that can be stored in the first part of the cache in some examples. In some examples, the repeating pattern of data includes a texture.

[0005] Another exemplary method includes the step of allocating cache entries to only a subset of cache lines of a data repeating pattern in response to a determination that all cache lines of a data repeating pattern exceed the storage capacity of the cache. In some examples, the method further includes the step of bypassing the storage of the remaining cache lines of the data repeating pattern in the cache. The step of allocating cache entries to only a subset includes, in some examples, the step of selecting cache lines of a data repeating pattern that exceed the maximum number of cache lines that the cache can store as a remainder.

[0006] In some examples, the method further comprises the steps of determining that the reuse distance of a cache line of a repeating pattern of data exceeds the number of cache lines that can be stored in the cache, and limiting a subset to the number of cache lines that can be stored in the cache. The method further comprises the steps of repeatedly selecting a cache line of a repeating pattern of data to be excluded from the cache, determining a hit rate for a cache line of a repeating pattern of data that is not excluded from the cache, and in some examples, selecting a cache line of a repeating pattern of data to be excluded from the cache based on the hit rate.

[0007] In some examples, the method further comprises the steps of dividing the cache into two or more parts and assigning the first part of the cache to a repeating pattern of data. The step of determining that the reuse distance of the cache lines of the repeating pattern of data exceeds the number of cache lines that can be stored in the cache includes the step of determining that the reuse distance of the cache lines of the repeating pattern of data exceeds the number of cache lines that can be stored in the first part of the cache. In some examples, the repeating pattern of data includes a texture.

[0008] In another example, the device includes a parallel processing unit. The parallel processing unit includes a cache and a cache controller that identify a repeating pattern of data having multiple cache lines exceeding the storage capacity of the cache and allocate storage to only a subset of the cache lines of the repeating pattern of data in the cache. In some examples, the cache controller bypasses storing the remaining cache lines of the repeating pattern of data in the cache.

[0009] In some examples, the cache controller is configured to select cache lines of a repeating pattern of data that exceed the maximum number of cache lines the cache can store as a remainder. The cache controller determines whether the reuse distance of the cache lines of the repeating pattern of data exceeds the number of cache lines that can be stored in the cache, and in some examples, is configured to limit a subset to the number of cache lines that can be stored in the cache. In one example, the cache controller is configured to repeatedly select cache lines of a repeating pattern of data to be excluded from the cache, determine a hit rate for cache lines of a repeating pattern of data that are not excluded from the cache, and select cache lines of a repeating pattern of data to be excluded from the cache based on the hit rate.

[0010] In another example, the cache controller is configured to divide the cache into two or more parts and assign a first part of the cache to a repeating pattern of data. The cache controller is further configured to determine that the reuse distance of a cache line of a first repeating pattern of data exceeds the number of cache lines that can be stored in the cache by determining that the reuse distance of a cache line of a first repeating pattern of data exceeds the number of cache lines that can be stored in the first part of the cache. Brief explanation of the drawing

[0011] The present disclosure may be better understood by those skilled in the art by referring to the accompanying drawings, and many features and advantages may become apparent. The use of the same reference symbol in different drawings indicates similar or identical items. FIG. 1 is a block diagram of a processing system configured to selectively cache a subset of repeating patterns of data according to some embodiments. FIG. 2 is a diagram illustrating caching a repeating pattern of data without selecting a subset of repeating patterns according to some embodiments. FIG. 3 is a diagram illustrating the caching of a repeating pattern of data according to some embodiments. FIG. 4 is a block diagram of a cache controller that allocates storage to the cache only for a subset of cache lines of a repeating pattern of data according to some embodiments. FIG. 5 is a block diagram of a cache controller that partitions a cache according to some embodiments and allocates storage to a portion of the cache only for a subset of cache lines of a repeating pattern of data. FIG. 6 is a diagram illustrating the selective caching of a cache line of a repeating pattern of data according to some embodiments. FIG. 7 is a diagram illustrating the selective caching of a cache line of a repeating pattern of data according to some embodiments. FIG. 8 is a flowchart illustrating a method for selectively caching a cache line of a repeating pattern of data according to some embodiments. Specific details for implementing the invention

[0012] The processor cores of a parallel processing unit often execute dispatches of work items that sequentially access repeating patterns of data, such as surfaces. For example, surfaces, such as textures, connected to a series of dispatched work items are read in successive rendering or denoising passes. For instance, many elements of a video game scene are rendered in successive frames as the position or angle of the camera view changes. Successive passes require the same texture data mapped to pixels. In some cases, the size of the texture exceeds the storage capacity of the parallel processing unit's lowest-level cache (L1 cache).

[0013] Generally, the L1 cache controller uses an LRU cache replacement policy and fetches data requested by processor cores that is not currently stored in the L1 cache (i.e., cache misses) from higher-level caches in the cache hierarchy. To make space for the requested cache line, the least-used cache line is removed from the L1 cache. However, if the reuse distance for a data repeat pattern (i.e., the number of cache lines in each repeat of the pattern) exceeds the storage capacity of the L1 cache, by the time a new request for a cache line in that repeat pattern is received by the cache controller, the cache line has already been removed from the cache, resulting in a low (or zero) hit rate. Retrieving data from more remote levels of the memory hierarchy to ensure access in subsequent dispatches consumes resources and increases latency.

[0014] FIGS. 1 through 8 illustrate a technique for a process system to selectively allocate storage to a local cache of a parallel processing unit for cache lines of a repeating pattern of data when the repeating pattern is of a size that exceeds the storage capacity of the cache. The processing system identifies a repeating pattern of data having cache lines with a reuse distance that exceeds the storage capacity of the cache. The cache controller allocates storage to only a subset (i.e., some, but not all) of the cache lines of the repeating pattern of data in the cache and excludes the remaining cache lines of the repeating pattern of data from the cache. For example, if the maximum number of cache lines that the cache can store at one time is X cache lines and the processing system identifies a repeating pattern of data that is X+Y cache lines, in some embodiments, the cache controller allocates storage to only the first X cache lines of the repeating pattern of data and excludes the remaining Y cache lines from being stored in the cache. In response to a request from a processor core for the remaining Y cache lines, the cache controller provides the remaining Y cache lines directly to the processor core at a higher level of the cache hierarchy and bypasses storing the Y cache lines in the cache. By restricting the cache to store only the first X cache line of the data's repeating pattern, the cache controller increases the cache hit rate for the first X cache line, thereby reducing latency and improving the user experience.

[0015] In some embodiments, the processing system probabilistically selects a subset of cache lines from a repeating pattern of data to be stored in the cache in a repetitive process. The processing system measures the hit rate for each selected subset of cache lines and selects the cache lines of the repeating pattern of data to be excluded from the cache based on the hit rate. In some embodiments, the cache controller divides the cache into two or more parts, allocates one part of the cache to store a subset of cache lines of a first repeating pattern of data, and allocates another part of the cache to store a subset of cache lines of a second repeating pattern of data. For the partitioned cache, the cache controller compares the reuse distance of the cache lines of the first repeating pattern of data with the storage capacity of the cache part allocated to the first repeating pattern of data. Based on the reuse distance and the storage capacity of the cache part, the cache controller selects the subset of cache lines of the first repeating pattern of data to be stored in the cache and excludes the remaining cache lines of the first repeating pattern of data from the cache.

[0016] FIG. 1 illustrates a processing system (100) configured to selectively allocate storage to a local cache of a parallel processing device for a cache line of a repeating pattern of data that exceeds the storage capacity of the cache according to some embodiments. The processing system (100) includes a parallel processing device (110), such as a graphics processing device (GPU) for generating a visual image intended to be output to a display (175), according to some embodiments. A parallel processor is a processor capable of executing a single instruction for multiple data or threads in a parallel manner.

[0017] Examples of parallel processors include processors such as graphics processing units (GPUs), superparallel processors, single instruction multiple data (SIMD) architecture processors, and single instruction multiple thread (SIMT) architecture processors for performing graphics, machine intelligence, or computing tasks. In some embodiments, the parallel processor is a separate device included as part of a computer. In other embodiments, such as an advanced processor unit, the parallel processor is included in a single device together with a host processor, such as a central processing unit (CPU). The description below uses a graphics processing unit (GPU) for illustrative purposes, but the embodiments and implementations described below are applicable to other types of parallel processors.

[0018] The processing system (100) includes a system memory (150) (referred to as memory (150) in this specification). Some embodiments of the memory (150) are implemented as dynamic random access memory (DRAM). However, the memory (150) may also be implemented using other types of memory, such as static random access memory (SRAM), non-volatile RAM, etc. In the illustrated embodiment, the parallel processing unit (110) communicates with the memory (150) via a bus (160). However, some embodiments of the parallel processing unit (110) communicate with the memory (150) via a direct connection or via another bus, bridge, switch, router, etc. The parallel processing unit (110) executes instructions stored in the memory (150), and the parallel processing unit (110) stores information, such as the result of the executed instructions, in the memory (150). For example, the memory (150) may store a copy of instructions from an application (155) to be executed by the parallel processing unit (110). Some embodiments of a parallel processing device (110) include a plurality of processor cores (referred to as computing devices) (115) that execute instructions independently, either simultaneously or in parallel.

[0019] A processing system (100) is generally configured to execute a set of instructions (e.g., a computer program), such as an application (155), to perform a specified task on an electronic device. Examples of such tasks include controlling the mode of operation of the electronic device, displaying information to a user to provide a specified user experience, communicating with other electronic devices, etc. Accordingly, in different embodiments, the processing system (100) is used in one of a number of types of electronic devices, such as a desktop computer, a laptop computer, a server, a game console, a tablet, a smartphone, etc. Components of the processing system (100) are implemented as hardware, firmware, software, or any combination thereof. It should be understood that the processing system (100) may include more or fewer components than illustrated in FIG. 1. For example, the processing system (100) may additionally include one or more input interfaces, non-volatile storage, one or more output interfaces, a network interface, and one or more displays or display interfaces.

[0020] The processing system (100) includes a central processing unit (CPU) (105) for executing instructions. Some embodiments of the CPU (105) include a plurality of processor cores (not shown for clarity) that execute instructions independently, either simultaneously or in parallel. The CPU (105) is also connected to a bus (160) and thus communicates with a parallel processing unit (110) and memory (150) through the bus (160). The CPU (105) executes instructions, such as program code for an application (155) stored in memory (150), and the CPU (105) stores information, such as the result of the executed instructions, in memory (150). The CPU (105) may also initiate graphics processing by issuing a draw call to the parallel processing unit (110). A draw call is a command generated by the CPU (105) and transmitted to the parallel processing unit (110) to instruct the parallel processing unit (110) to render an object within a frame (or part of an object). Some embodiments of a draw call include information defining textures, states, shaders, rendering objects, buffers, etc., used by the parallel processing unit (110) to render an object or part thereof. The parallel processing unit (110) renders the object to generate pixel values ​​provided to the display (175), and the display uses the pixel values ​​to display an image representing the rendered object.

[0021] In some embodiments, each frame to be rendered is processed by a graphics pipeline of a parallel processing unit (110) in multiple passes. For example, during the first pass regarding scene geometry, only the attributes necessary to calculate per-pixel lighting are written to the G-buffer. During the second pass, the graphics pipeline outputs only diffuse and reflective lighting data. In the third pass of the frame through the graphics pipeline, the graphics pipeline reads the lighting data again and outputs the final per-pixel shading. Thus, in multi-pass rendering, the associated objects of the scene and frame are rendered multiple times. Each time an object is drawn, the graphics pipeline calculates additional aspects of the object's appearance and combines the additional aspects with the previous results. Each time a frame or an object of a frame is rendered by the graphics pipeline is referred to as a render pass.

[0022] The input / output (I / O) engine (170) handles input or output operations associated with other elements of the processing system (100), such as a keyboard, mouse, printer, external disk, as well as a display (175). The I / O engine (170) is coupled to a bus (160) to allow the I / O engine (170) to communicate with a parallel processing unit (110), memory (150), or CPU (105). In the illustrated embodiment, the I / O engine (170) is configured to read information stored on an external storage medium (180), such as a compact disc (CD) or a digital video disc (DVD). The external storage medium (180) stores information representing program code used to implement applications such as video games. The program code on the external storage medium (180) can be written to memory (150) to form a copy (125) of instructions to be executed by the parallel processing unit (110) or CPU (105).

[0023] In some embodiments, a parallel processing unit (110) implements a graphics pipeline (not shown in FIG. 1 for clarity) comprising a plurality of stages configured for the simultaneous processing of different primitives in response to a draw call. The stages of the graphics pipeline of the parallel processing unit (110) can simultaneously process different primitives generated by an application such as a video game. When geometry is submitted to the graphics pipeline, a hardware state setting is selected to define the state of the graphics pipeline. Examples of states include a rasterizer state used to render the scene, a blend state, a depth stencil state, a primitive topology type of the submitted geometry, and a shader ( for example It includes vertex shaders, domain shaders, geometry shaders, hull shaders, pixel shaders, etc. Shaders implemented in the graphics pipeline state are represented by corresponding bytecode. In some cases, information representing the graphics pipeline state is hashed or compressed to provide a more efficient representation of the graphics pipeline state.

[0024] A driver (165) is a computer program that enables a higher-level graphics computing program, such as one from an application (155), to interact with a parallel processing unit (110). For example, the driver (165) converts standard code received from the application (155) into a native format command stream understood by the parallel processing unit (110). The driver (165) accepts input from the application (155) as direct settings of the parallel processing unit (110). These settings include the selection of render mode, anti-aliasing control, texture filter control, batch binning control, and deferred pixel shading control.

[0025] To execute a set of instructions received from a CPU, a parallel processing unit (110) includes a plurality of computing units (115) and an L1 cache (120). The plurality of computing units (115) together perform shading operations for the dispatch of work items (not shown). In different embodiments, the computing unit (115) performs geometry operations, texture operations, tessellation operations, vertex operations, mesh operations, primitive operations, ray tracing operations, computing operations, etc., or any combination thereof, based on instructions received from an instruction processor (not shown). In some embodiments, to perform these operations, the computing unit (115) includes one or more SIMD elements configured to execute a specified operation using the work item of the received dispatch.

[0026] The L1 cache (120) stores data for multiple computing devices (115). Thus, during the execution of shader operations, the multiple computing devices (115) store and retrieve data from the L1 cache (120), whereby the stored and retrieved data is based on a specific task item being processed. For example, in some embodiments, each task item of a dispatch corresponds to an individual pixel of an image, and the L1 cache (120) stores data (e.g., texture values) for each individual pixel included in the dispatch, or a subset of individual pixels. In some embodiments, the parallel processing device (110) is associated with a memory hierarchy having multiple cache levels as well as system memory (150), and the L1 cache (120) represents the lowest level among the multiple cache levels.

[0027] To increase the hit rate in the L1 cache (120) for a repeat pattern of data that exceeds the storage capacity of the L1 cache (120), the parallel processing unit (110) includes a cache controller (130) configured to allocate storage only for a subset (140) of cache lines of the repeat pattern of data in the L1 cache (120) and exclude the remaining cache lines of the repeat pattern of data from the cache. In response to a request from a computing unit (115) for the remaining cache lines of the repeat pattern of data, the cache controller (130) provides the remaining cache lines directly to the computing unit(s) (115) from a higher level of the cache hierarchy (not shown) and bypasses storing the remaining cache lines in the L1 cache (120). By limiting the L1 cache (120) to simultaneously store only cache lines of repeating data patterns that fit within the storage capacity of the L1 cache (120), the cache controller (130) increases the hit rate in the L1 cache (120) for a selected subset (140) of cache lines.

[0028] The cache controller (130) includes a pattern recognition unit (135) configured to identify a repeating pattern of data that exceeds the storage capacity of the L1 cache (120). The pattern recognition unit (135) measures the reuse distance of a cache line in the repeating pattern of data (i.e., the number of multiple cache lines in each repetition of the pattern) and compares the reuse distance (DR) with the total number of cache lines that the L1 cache (120) can store CL1 (i.e., the storage capacity of the L1 cache (120)). If the reuse distance (DR) exceeds the storage capacity (CL1) of the L1 cache (120), the cache controller (130) limits the number of cache lines of the repeating pattern of the data allocated in the storage space of the L1 cache (120) to a subset (140) of cache lines that fits within the storage capacity (CL1) of the L1 cache (120). The cache controller (130) excludes the remaining DR - CL1 cache lines of the repeating pattern of data from the L1 cache (120). In some embodiments, in response to an L1 cache (120) exhibiting a relatively low hit rate, the pattern recognition unit (135) probabilistically selects a subset of cache lines from a repeating pattern of data to be stored in the L1 cache (120) in an iterative process. The cache controller (130) measures the hit rate for each selected subset of cache lines and selects the cache lines of the repeating pattern of data to be excluded from the cache based on the hit rate.

[0029] In some embodiments, each draw call is associated with an identifier (not shown), and a request stream for a series of dispatches of work items includes a draw call identifier and a surface on which the draw call will operate. A pattern recognition unit (135) tracks the number of requests for draw call identifier / surface combinations excluded from the L1 cache (120). The pattern recognition unit (135) lists draw call identifier / surface combinations according to the frequency of exclusion from the L1 cache (120). In some embodiments, a cache controller (130) simulates a hit rate for a selected subset of cache lines (140) based on the inclusion or exclusion of draw call identifier / surface combinations from the L1 cache (120) to determine which draw call identifier / surface combinations to exclude from the L1 cache (120) to increase the hit rate.

[0030] In some embodiments, the cache controller (130) divides the L1 cache (120) into two or more parts (not shown), allocates one part of the L1 cache (120) to store a subset of cache lines of a first repeat pattern of data, and allocates another part of the L1 cache (120) to store a subset of cache lines of a second repeat pattern of data. For the divided L1 cache (120), the cache controller (130) compares the reuse distance (DR) of the cache lines of the first repeat pattern of data with the storage capacity of the cache part to which the first repeat pattern of data is allocated. Based on the reuse distance (DR) and the storage capacity of the cache part, the cache controller (130) selects a subset of cache lines of the first repeat pattern of data to be stored in the cache and excludes the remaining cache lines of the first repeat pattern of data from the cache.

[0031] FIG. 2 is a diagram (200) illustrating caching a repeating pattern of data without selecting a subset of cache lines based on reuse distance according to some embodiments. In the illustrated example, the L1 cache (120) is sized to hold up to three cache lines at a time. After the L1 cache (120) is flushed, at time (T1), the cache controller (130) receives a request for cache line-1 (202) from the computing device (115). Since the L1 cache (120) does not contain valid data, the request results in a miss in the L1 cache (120), and the cache controller (130) copies cache line-1 (202) from the L2 cache (220), which is a higher-level cache in the cache hierarchy, to the L1 cache (120) and provides cache line-1 (202) to the computing device (115). At a subsequent time (T2), the cache controller (130) receives a request for cache line-2 (204) from the computing device (115). Since the L1 cache (120) does not contain cache line-2 (204), the request results in a missing cache in the L1 cache (120), and the cache controller (130) copies cache line-2 (204) from the L2 cache (220) to the L1 cache (120) and provides cache line-2 (204) to the computing device (115). At a subsequent time (T3), the cache controller receives a request for cache line-3 (206) from the computing device (115). Since the L1 cache (120) does not contain cache line-3 (206), the request results in a missing cache in the L1 cache (120), and the cache controller (130) copies cache line-3 (206) from the L2 cache (220) to the L1 cache (120) and provides cache line-3 (206) to the computing device (115).

[0032] At a subsequent time (T4), the cache controller (130) receives a request for cache line-4 (208) from the computing device (115). Since the L1 cache (120) does not contain cache line-4 (208), the request results in a missing cache in the L1 cache (120). To make space for cache line-4 (208) in the L1 cache (120), the cache controller removes cache line-1 (202), which is the least used cache line in the current situation. The cache controller (130) copies cache line-4 (208) from the L2 cache (220) to the L1 cache (120) and provides cache line-4 (208) to the computing device (115).

[0033] At a subsequent time (T5), the cache controller (130) receives a request for cache line-5 (210) from the computing device (115). Since the L1 cache (120) does not contain cache line-5 (210), the request results in a missing cache in the L1 cache (120). To make space for cache line-5 (208) in the L1 cache (120), the cache controller removes cache line-2 (204), which is the least used cache line in the current situation. The cache controller (130) copies cache line-5 (210) from the L2 cache (220) to the L1 cache (120) and provides cache line-5 (210) to the computing device (115). Therefore, during the first 5 cycles (T1 to T5) of access requests for cache lines (202, 204, 206, 208, 210), the cache hit rate is 0%.

[0034] FIG. 3 is a diagram (300) illustrating the continuation of the caching of the repeating pattern of the data of FIG. 2. At a subsequent time (T6), the cache controller (130) receives a request for cache line-1 (202) from the computing device (115). Since the L1 cache (120) does not contain cache line-1 (202) (because cache line-1 (202) was previously removed at time (T4), the request results in a missing cache in the L1 cache (120). To make space for cache line-1 (202) in the L1 cache (120), the cache controller (130) removes cache line-3 (206), which is the least-used cache line in the current situation. The cache controller (130) copies cache line-1 (202) from the L2 cache (220) to the L1 cache (120) and provides cache line-1 (202) to the computing device (115).

[0035] At a subsequent time (T7), the cache controller (130) receives a request for cache line-2 (204) from the computing device (115). Since the L1 cache (120) does not contain cache line-2 (204) (because cache line-2 (202) was previously removed at time (T5), the request results in a missing cache in the L1 cache (120). To make space for cache line-2 (204) in the L1 cache (120), the cache controller (130) removes cache line-4 (208), which is the least used cache line in the current situation. The cache controller (130) copies cache line-2 (204) from the L2 cache (220) to the L1 cache (120) and provides cache line-2 (204) to the computing device (115).

[0036] At a subsequent time (T8), the cache controller (130) receives a request for cache line-3 (206) from the computing device (115). Since the L1 cache (120) does not contain cache line-3 (206) (because cache line-3 (206) was previously removed at time (T6), the request results in a missing cache in the L1 cache (120). To make space for cache line-3 (206) in the L1 cache (120), the cache controller (130) removes cache line-5 (210), which is the least used cache line in the current situation. The cache controller (130) copies cache line-3 (206) from the L2 cache (220) to the L1 cache (120) and provides cache line-3 (206) to the computing device (115).

[0037] At a subsequent time (T9), the cache controller (130) receives a request for cache line-4 (208) from the computing device (115). Since the L1 cache (120) does not contain cache line-4 (208) (because cache line-4 (208) was previously removed at time (T7), the request results in a missing cache in the L1 cache (120). To make space for cache line-4 (208) in the L1 cache (120), the cache controller removes cache line-1 (202), which is the least used cache line in the current situation. The cache controller (130) copies cache line-4 (208) from the L2 cache (220) to the L1 cache (120) and provides cache line-4 (208) to the computing device (115).

[0038] At a subsequent time (T10), the cache controller (130) receives a request for cache line-5 (210) from the computing device (115). Since the L1 cache (120) does not contain cache line-4 (208) (because cache line-5 (210) was previously removed at time (T8), the request results in a missing cache in the L1 cache (120). To make space for cache line-5 (210) in the L1 cache (120), the cache controller removes cache line-2 (204), which is the least used cache line in the current situation. The cache controller (130) copies cache line-5 (210) from the L2 cache (220) to the L1 cache (120) and provides cache line-5 (210) to the computing device (115). Therefore, for a repetitive 5-cache-line pattern of data in cache lines (202, 204, 206, 208, 210), caching using an LRU replacement policy results in a 0% hit rate for the L1 cache (120) during cycles (T6 to T10), and the hit rate of the cold start scenario exemplified in FIG. 2 does not improve to 0% or higher.

[0039] FIG. 4 is a block diagram (400) of a cache controller (130) of FIG. 1 that allocates storage to the L1 cache (120) only for a subset (140) of cache lines of a repeating pattern of data (420) according to some embodiments. In the example illustrated, the L1 cache (120) is sized to hold up to 3 cache lines at a time. Since the repeating pattern of data (420) includes 5 cache lines (1, 2, 3, 4, 5, 1, 2, 3, 4, 5, 1, 2, 3, 4, 5, ...), it exceeds the storage capacity of the L1 cache (120). The cache controller (130) uses a cache replacement policy (405) that removes the least-used-frequency cache line when the L1 cache (120) is full and cache lines are fetched from a higher-level cache.

[0040] The pattern recognition unit (135) identifies a repeating pattern of data (420) as data is retrieved from GPU memory (410) and determines that the cache line of the repeating pattern of data (420) has a reuse distance that exceeds the storage capacity of the L1 cache (120). As the pattern recognition unit (135) determines that the cache line of the repeating pattern of data (420) has a reuse distance that exceeds the storage capacity of the L1 cache (120), the cache control unit (130) allocates storage only to a subset (140) of cache lines (e.g., cache lines 1, 2, and 3) that can be stored in the L1 cache (120) at once, and excludes the remaining cache lines (415) (cache lines 4 and 5) of the repeating pattern of data (420) from the L1 cache (120). Accordingly, when the computing device (115) requests each of the cache lines (1, 2, and 3) of the subset (140) of cache lines, the cache controller allocates an entry for the cache lines (1, 2, and 3) and stores it in the L1 cache (120). However, when the computing device (115) requests the remaining cache lines (415) (cache lines 4 and 5), the cache controller provides the remaining cache lines (415) directly from the L2 cache (220) to the computing device (115) and bypasses storing the remaining cache lines (415) in the L1 cache (120).

[0041] FIG. 5 is a block diagram (500) of a cache controller (130) that partitions an L1 cache (120) according to some embodiments and allocates storage to a portion of the cache only for a subset of cache lines of repeating patterns of data. Sometimes, a parallel processing unit (110) handles draw calls that access more than one texture in parallel. In the illustrated example, the cache controller (130) retrieves two textures from GPU memory (410) for nested access, namely a first repeating pattern of data (520) and a second repeating pattern of data (530). To increase the hit rate for the first repeating pattern of data (520) and the second repeating pattern of data (530), the cache controller (130) partitions the L1 cache (120) into a first portion (540) and a second portion (545).

[0042] In response to the pattern recognition unit (135) identifying a first repeating pattern of data (520) and determining that the reuse distance (550) for the cache line of the first repeating pattern of data (520) exceeds the storage capacity (555) of the first part (540) of the L1 cache, the cache control unit (130) allocates storage to the first part (540) to store a subset of the cache line (525) of the first repeating pattern of data (520). Likewise, in response to the pattern recognition unit (135) identifying a second repeating pattern of data (530) and determining that the reuse distance (560) for the cache line of the second repeating pattern of data (530) exceeds the storage capacity (565) of the second part (545) of the L1 cache, the cache control unit (130) allocates storage to the second part (545) to store a subset of the cache line (535) of the second repeating pattern of data (530).

[0043] FIG. 6 is a diagram (600) illustrating the selective caching of cache lines of a repeating pattern of data according to some embodiments. Similar to the diagram (200) illustrated in FIG. 2, the repeating pattern of data includes five cache lines, whereas the L1 cache (120) has a storage capacity of three cache lines. However, unlike the diagram (200) illustrated in FIG. 2, the cache controller (130) implements selective caching for the repeating pattern of data in the illustrated example. As the pattern recognition unit (135) determines that the cache lines of the repeating pattern of data have a reuse distance that exceeds the storage capacity of the L1 cache (120), the cache controller allocates space within the L1 cache (120) only for a subset of the cache lines of the repeating pattern of data (cache lines (202, 204, and 206)) and excludes the remaining cache lines of the repeating pattern of data from the L1 cache (120).

[0044] After the L1 cache (120) is flushed, at time (T1), the cache controller (130) receives a request for cache line-1 (202) from the computing device (115). Since the L1 cache (120) does not contain valid data, the request results in a missing from the L1 cache (120). Since cache line-1 (202) is included in a subset of cache lines for which the cache controller (130) has allocated storage to the L1 cache (120), the cache controller (130) copies cache line-1 (202) from the L2 cache (220) to the L1 cache (120) and provides cache line-1 (202) to the computing device (115).

[0045] At a subsequent time (T2), the cache controller (130) receives a request for cache line-2 (204) from the computing device (115). Since the L1 cache (120) does not contain cache line-2 (204), the request results in a missing cache in the L1 cache (120). Since the cache controller (130) has included cache line-2 (204) in a subset of cache lines for which storage is allocated in the L1 cache (120), the cache controller (130) copies cache line-2 (204) from the L2 cache (220) to the L1 cache (120) and provides cache line-2 (204) to the computing device (115).

[0046] At a subsequent time (T3), the cache controller receives a request for cache line-3 (206) from the computing device (115). Since the L1 cache (120) does not contain cache line-3 (206), the request results in a missing cache in the L1 cache (120). Since the cache controller (130) has included cache line-3 (206) in a subset of cache lines for which storage is allocated in the L1 cache (120), the cache controller (130) copies cache line-3 (206) from the L2 cache (220) to the L1 cache (120) and provides cache line-3 (206) to the computing device (115).

[0047] At a subsequent time (T4), the cache controller (130) receives a request for cache line-4 (208) from the computing device (115). Since the L1 cache (120) does not contain cache line-4 (208), the request results in a missing cache in the L1 cache (120). Because the cache controller (130) did not include cache line-4 (208) in the subset of cache lines of the repeating pattern of data allocated storage in the L1 cache (120), the cache controller provides cache line-4 (208) directly from the L2 cache (220) to the computing device (115) and excludes cache line-4 (208) from the L1 cache (120).

[0048] At a subsequent time (T5), the cache controller (130) receives a request for cache line-5 (210) from the computing device (115). Since the L1 cache (120) does not contain cache line-5 (210), the request results in a missing cache in the L1 cache (120). Because the cache controller (130) did not include cache line-5 (210) in the subset of cache lines of the repeating pattern of data for which storage is allocated in the L1 cache (120), the cache controller provides cache line-5 (210) directly from the L2 cache (220) to the computing device (115) and excludes cache line-5 (210) from the L1 cache (120). Accordingly, similar to the illustrated figure (200) in FIG. 2, during the first 5 cycles (T1 to T5) of access requests for cache lines (202, 204, 206, 208, 210), the cache hit rate is 0%.

[0049] FIG. 7 is a diagram (700) illustrating the successive selective caching of the repeating pattern of the data of FIG. 6. At a subsequent time (T6), the cache controller (130) receives a request for cache line-1 (202) from the computing device (115). Since the L1 cache (120) contains cache line-1 (202), the request causes a hit in the L1 cache (120), and the cache controller (130) provides cache line-1 (202) to the computing device (115). At a subsequent time (T7), the cache controller (130) receives a request for cache line-2 (204) from the computing device (115). Since the L1 cache (120) contains cache line-2 (204), the request hits the L1 cache (120), and the cache controller (130) provides cache line-2 (204) to the computing device (115). At a subsequent time (T8), the cache controller (130) receives a request for cache line-3 (206) from the computing device (115). Since the L1 cache (120) contains cache line-3 (206), the request hits the L1 cache (120), and the cache controller (130) provides cache line-3 (206) to the computing device (115).

[0050] At a subsequent time (T9), the cache controller (130) receives a request for cache line-4 (208) from the computing device (115). Since the cache controller (130) has excluded cache line-4 (208) from the L1 cache (120), the L1 cache (120) does not contain cache line-4 (208). Therefore, the request results in a missing cache in the L1 cache (120). Since the cache controller (130) does not include cache line-4 (208) in the subset of cache lines for which space is allocated in the L1 cache (120), it retrieves cache line-4 (208) from the L2 cache (220) and provides cache line-4 (208) directly to the computing device (115), bypassing the storage of cache line-4 (208) in the L1 cache (120).

[0051] At a subsequent time (T10), the cache controller (130) receives a request for cache line-5 (210) from the computing device (115). Since the cache controller (130) has excluded cache line-5 (210) from the L1 cache (120), the L1 cache (120) does not contain cache line-5 (210). Therefore, the request results in a missing cache in the L1 cache (120). Since the cache controller (130) does not include cache line-5 (210) in the subset of cache lines for which space is allocated in the L1 cache (120), it retrieves cache line-5 (210) from the L2 cache (220) and provides cache line-5 (210) directly to the computing device (115), bypassing the storage of cache line-5 (210) in the L1 cache (120). Accordingly, for a repetitive 5-cache-line pattern of data in cache lines (202, 204, 206, 208, 210), selective caching using an LRU replacement policy results in a 60% hit rate for the L1 cache (120) during cycles (T6 to T10), which is a significant increase from the 0% hit rate of the conventional caching scenario exemplified in FIG. 3. Additionally, additional cycles of the same repetitive pattern of data can be expected to result in a similarly increased hit rate for the L1 cache (120) using selective caching.

[0052] FIG. 8 is a flowchart illustrating a method (800) for selectively caching a cache line of a repeating pattern of data according to some embodiments. In some embodiments, the method (800) is performed by a processing system such as the processing system (100) illustrated in FIG. 1. In block (802), a pattern recognition unit (135) identifies a repeating pattern of data (420). In block (804), the pattern recognition unit (135) compares the reuse distance of a cache line of a repeating pattern of data (420) with the maximum number of cache lines that an L1 cache (120) can store at once, and determines whether the reuse distance exceeds the storage capacity of the L1 cache (120).

[0053] In block (804), if the pattern recognition unit (135) determines that the reuse distance of the cache lines of the repeating pattern of data (420) does not exceed the storage capacity of the L1 cache (120), the method flow continues to block (806). In block (806), the cache controller (130) caches all cache lines of the repeating pattern of data in the L1 cache (120).

[0054] In block (804), if the pattern recognition unit (135) determines that the reuse distance of the cache lines of the repeating pattern of data (420) exceeds the storage capacity of the L1 cache (120), the method flow continues to block (808). In block (808), the cache controller (130) allocates storage to the L1 cache (120) only for a subset (140) of the cache lines of the repeating pattern of data (420) and excludes the remaining cache lines (415) of the repeating pattern of data (420) from the L1 cache (120). In some embodiments, the pattern recognition unit (135) probabilistically selects a subset of cache lines from the repeating pattern of data to be stored in the L1 cache (120) in a repeating process. The cache controller (130) measures the hit rate for each subset of selected cache lines and, based on the hit rate, selects a subset (140) of cache lines of repeating patterns of data (420) to be excluded from the L1 cache (120).

[0055] In some embodiments, the cache controller (130) divides the L1 cache (120) into a first portion (540) and a second portion (545). In response to the pattern recognition unit (135) identifying a first repeating pattern of data (520) and determining that the reuse distance for the cache line of the first repeating pattern of data (520) exceeds the storage capacity of the first portion (540) of the L1 cache, the cache controller (130) allocates storage to the first portion (540) to store a subset of the cache line (525) of the first repeating pattern of data (520).

[0056] In some embodiments, the devices and technologies described above are implemented in a system comprising one or more integrated circuit (IC) devices (also referred to as integrated circuit packages or microchips), such as the processing system described with reference to FIGS. 1 through 8. Electronic design automation (EDA) and computer-aided design (CAD) software tools may be used for the design and manufacture of these IC devices. These design tools are typically represented as one or more software programs. One or more software programs include code executable by a computer system to operate the computer system to operate on code representing a circuit of one or more IC devices to perform at least part of a process for designing or adapting a manufacturing system to manufacture the circuit. This code may include instructions, data, or a combination of instructions and data. Software instructions representing design tools or manufacturing tools are typically stored on a computer-readable storage medium accessible to the computing system. Likewise, code representing one or more steps of the design or manufacture of the IC device may be stored and accessed on the same computer-readable storage medium or another computer-readable storage medium.

[0057] Computer-readable storage media may include any non-transient storage media or combination of non-transient storage media that a computer system can access during use to provide instructions and / or data to a computer system. Such storage media may include, but are not limited to, optical media (e.g., Compact Disc (CD), Digital Multifunction Disc (DVD), Blu-ray Disc), magnetic media (e.g., floppy disk, magnetic tape, or magnetic hard drive), volatile memory (e.g., Random Access Memory (RAM) or cache), non-volatile memory (e.g., Read-Only Memory (ROM) or flash memory), or microelectromechanical systems (MEMS)-based storage media. Computer-readable storage media may be embedded in a computing system (e.g., system RAM or ROM), fixedly attached to a computing system (e.g., magnetic hard drive), detachably attached to a computing system (e.g., optical disc or Universal Serial Bus (USB)-based flash memory), or connected to a computer system via a wired or wireless network (e.g., Network Accessible Storage (NAS)).

[0058] In some embodiments, specific aspects of the described technology may be implemented by one or more processors of a processing system executing software. The software comprises one or more executable instruction sets stored on a non-transient computer-readable storage medium or otherwise implemented as a type. When executed by one or more processors, the software may include instructions and specific data that manipulate one or more processors to perform one or more aspects of the technology described above. The non-transient computer-readable storage medium may include, for example, magnetic or optical disk storage devices, solid-state storage devices such as flash memory, caches, random access memory (RAM), or other non-volatile memory devices or devices. Executable instructions stored on the non-transient computer-readable storage medium may be source code, assembly language code, object code, or other instruction formats that are interpreted or otherwise executable by one or more processors.

[0059] It should be noted that in the general description, not all activities or elements described above are required, and parts of specific activities or devices may not be required, and one or more additional activities may be performed or elements may be included in addition to those described above. Furthermore, the order in which activities are listed does not necessarily correspond to the order in which they are performed. Additionally, the concept has been described with reference to specific embodiments. However, those skilled in the art will understand that various modifications and changes are possible without departing from the scope of the disclosure, as set forth in the claims below. Accordingly, the specification and drawings should be considered illustrative rather than restrictive, and all such modifications are intended to be included within the scope of the disclosure.

[0060] Advantages, other advantages, and solutions to problems have been described above in relation to specific embodiments. However, any feature(s) that may give rise to or make more apparent any advantage, advantage, or solution, and any other advantage, advantage, or solution, should not be interpreted as important, required, or essential features of any or all claims. Furthermore, the specific embodiments disclosed above are merely illustrative, and the subject matter disclosed may be modified and practiced in different but equivalent ways that are obvious to those skilled in the art having the benefit of the teachings of this specification. No limitation is intended on the details of the configuration or design described in this specification other than those set forth in the following claims. Accordingly, it is evident that the specific embodiments disclosed above may be altered or modified, and that all such variations are considered within the scope of the subject matter disclosed. Accordingly, the scope of protection sought in this specification is as set forth in the claims below.

Claims

Claim 1 A method comprising the step of allocating storage to the cache by a cache controller only for a subset of the cache lines of said single instance in response to said single instance of a pattern of repetitive data spanning multiple cache lines that exceed the storage capacity of the cache in which said pattern of repetitive data is to be stored. Claim 2 A method according to claim 1, further comprising the step of bypassing the storage of the remaining cache lines of the single instance in the cache. Claim 3 In paragraph 2, the step of allocating storage only to the subset comprises selecting cache lines of the pattern of the repeating data that exceed the maximum number of cache lines that the cache can store as the remaining cache lines. Claim 4 A method according to claim 1, further comprising: determining that the reuse distance of a cache line of the single instance exceeds the number of cache lines that can be stored in the cache; and limiting the subset to the number of cache lines that can be stored in the cache. Claim 5 A method according to claim 4, further comprising the step of repeatedly selecting cache lines of the pattern of the repeating data to be excluded from the cache, wherein a hit rate is determined for the subset of cache lines of the pattern of the repeating data not excluded from the cache; and by selecting cache lines of the pattern of the repeating data to be excluded from the cache based on the hit rate, the cache lines of the pattern of the repeating data to be excluded from the cache are repeatedly selected. Claim 6 A method according to claim 4, further comprising: a step of dividing the cache into two or more parts; and a step of assigning a first part of the cache to a pattern of the repeated data of the data, wherein the step of determining that the reuse distance of a cache line of the single instance exceeds the number of cache lines that can be stored in the cache includes the step of determining that the reuse distance of a cache line of the single instance exceeds the number of cache lines that can be stored in the first part of the cache. Claim 7 A method according to claim 1, wherein the pattern of the repeating data includes a texture. Claim 8 A device comprising: a parallel processing unit including a cache to store a pattern of repeating data; and a cache controller, wherein the cache controller is configured to allocate storage to the cache only for a subset of the cache lines of a single instance in response to a single instance of the pattern of repeating data having a plurality of cache lines that exceed the storage capacity of the cache. Claim 9 In claim 8, the cache controller is a device that bypasses storing the remaining cache lines of the repeating data pattern in the cache. Claim 10 In claim 9, the device is configured such that the cache controller selects cache lines of the pattern of the repeating data that exceed the maximum number of cache lines that the cache can store as the remaining cache lines. Claim 11 In claim 8, the cache controller determines whether the reuse distance of a cache line of the single instance exceeds the number of cache lines that can be stored in the cache; and the device is configured to limit the subset to the number of cache lines that can be stored in the cache. Claim 12 In claim 11, the cache controller determines a hit rate for cache lines of the pattern of the repetitive data that are not excluded from the cache; and selects cache lines of the pattern of the repetitive data to be excluded from the cache based on the hit rate, the device. Claim 13 A device according to claim 11, wherein the cache controller divides the cache into two or more parts; assigns a first part of the cache to a pattern of the repetitive data; and determines that the reuse distance of a cache line of a single instance exceeds the number of cache lines that can be stored in the first part of the cache, thereby determining that the reuse distance of a cache line of a single instance exceeds the number of cache lines that can be stored in the cache. Claim 14 In claim 8, the pattern of the repeating data comprises a texture, a device. Claim 15 delete Claim 16 delete Claim 17 delete Claim 18 delete Claim 19 delete Claim 20 delete

Images