Intermediate cache management for non-uniform memory architecture

Abstract

A cache controller (104) of a processing system (100) implementing a non-uniform memory architecture (NUMA) adjusts cache replacement priorities of local data and non-local data stored at a cache based on a cache replacement policy (112). The local data (326) is data accessed by the cache (102) via a local memory channel (106), and the non-local data (324) is data accessed by the cache (102) via a non-local memory channel (116). The cache controller assigns priorities to local data and non-local data stored at the cache based on a cache replacement policy, and selects data for replacement at the cache based at least in part on the assigned priorities.

Description

BACKGROUND

[0001] Processing systems often include a memory subsystem having memory modules to store data to be accessed by instructions being executed. The memory subsystem can be organized into a memory hierarchy having a main memory at the top of the hierarchy to store a larger amount of data that can be accessed by executing instructions, and one or more caches at lower levels of the memory hierarchy to store a subset of the data stored at the main memory. Generally, the lower in the memory hierarchy that data is stored, the faster the processor can access that data. To further enhance processing efficiency, processing systems can implement a memory management protocol that manages particular sets of data stored at each level of the memory hierarchy. For example, a processing system can implement a memory management protocol that moves data that has recently been requested for access to a lower level of the memory hierarchy, expects that the data will be accessed again by the processor in the near future, and moves data that has not recently been accessed to a higher level of the memory hierarchy. SUMMARY

[0002] Examples described herein illustrate techniques for a processing system to adjust cache replacement priorities of local data and non-local data stored at a cache based on a cache replacement policy. In some implementations, the processing system includes a plurality of chiplet subsets, and the cache is an intermediate cache located at a first chiplet subset. In a first example, a method includes assigning, based on a cache replacement policy, a priority of local data and a priority of non-local data, the local data stored at a cache at a first chiplet subset of a processing system including a plurality of chiplet subsets, wherein the local data includes data accessed from a local memory at the first chiplet subset via a local memory channel, the non-local data stored at the cache, wherein the non-local data includes data accessed from a non-local memory at a second chiplet subset of the processing system via a non-local memory channel. The method further includes selecting data for replacement at the cache based on the assigned priorities.

[0003] In some implementations, the method further includes periodically measuring an amount of local data stored at the cache and an amount of non-local data stored at the cache. The method can further include adjusting the priority of the local data stored at the cache and the priority of the non-local data stored at the cache based on a ratio of the local data to the non-local data stored at the cache.

[0004] In some implementations, the method further includes increasing the priority of the non-local data in response to the amount of non-local data stored at the cache exceeding the amount of local data stored at the cache by more than a first threshold amount. In other implementations, the method further includes increasing the priority of the local data in response to the amount of local data stored at the cache exceeding the amount of non-local data stored at the cache by more than a second threshold amount.

[0005] The method can further include assigning equal priority to the local data and the non-local data stored at the cache in response to the amount of non-local data stored at the cache not exceeding the amount of local data stored at the cache by the first threshold amount and the amount of local data stored at the cache not exceeding the amount of non-local data stored at the cache by more than the second threshold amount. In some implementations, assigning the respective priorities of the local data and the non-local data is based on an application program executing at the processing system.

[0006] In another example, a processing system includes a cache at a first subset of dies of the processing system, the cache configured to store local data accessed from local memory at the first subset of dies via a local memory channel and non-local data accessed from non-local memory at a second subset of dies of the processing system via a non-local memory channel. The processing system further includes a cache controller configured to assign a priority to the local data and to assign a priority to the non-local data based on a cache replacement policy and to select data for replacement at the cache based on the assigned priorities.

[0007] In some implementations, the cache controller periodically measures an amount of local data stored at the cache and an amount of non-local data stored at the cache. The cache controller can further adjust the priority of the local data stored at the cache and the priority of the non-local data stored at the cache based on a ratio of the local data to the non-local data stored at the cache.

[0008] For example, in some implementations, the cache controller is configured to increase the priority of the non-local data in response to the amount of non-local data stored at the cache exceeding the amount of local data stored at the cache by more than a first threshold amount. In other implementations, the cache controller is configured to increase the priority of the local data in response to the amount of local data stored at the cache exceeding the amount of non-local data stored at the cache by more than a second threshold amount.

[0009] The cache controller can be configured to assign equal priority to local data and non-local data stored at the cache in response to an amount of non-local data stored at the cache not exceeding an amount of local data stored at the cache by a first threshold amount and an amount of local data stored at the cache not exceeding an amount of non-local data stored at the cache by more than a second threshold amount. In some implementations, the cache controller is further configured to assign respective priorities to the local data and the non-local data based on an application executing at the processing system.

[0010] In another example, a processing system includes a cache at a first subset of dies of the processing system and a cache controller configured to adaptively prioritize one of local data and non-local data for replacement at the cache based on a cache replacement policy and a ratio of the local data and the non-local data stored at the cache, where the local data includes data accessed from local memory at the first subset of dies via a local memory channel and the non-local data includes data accessed from non-local memory at a second subset of dies of the processing system via a non-local memory channel.

[0011] The cache controller can be further configured to periodically measure an amount of local data stored at the cache and an amount of non-local data stored at the cache. In some implementations, the cache controller is further configured to increase a priority of the non-local data in response to an amount of non-local data stored at the cache exceeding an amount of local data stored at the cache by more than a first threshold amount. In other implementations, the cache controller is further configured to increase a priority of the local data in response to an amount of local data stored at the cache exceeding an amount of non-local data stored at the cache by more than a second threshold amount.

[0012] The cache controller can be configured to assign equal priority to local data and non-local data stored at the cache in response to an amount of non-local data stored at the cache not exceeding an amount of local data stored at the cache by a first threshold amount and an amount of local data stored at the cache not exceeding an amount of non-local data stored at the cache by more than a second threshold amount. Further, in some implementations, the cache controller is further configured to assign respective priorities to the local data and the non-local data based on an application executing at the processing system. BRIEF DESCRIPTION OF DRAWINGS

[0013] The disclosure can be better understood with reference to the following drawings and detailed description. The use of the same reference numbers in different drawings indicates the same or similar items.

[0014] Figure 1 is a block diagram of a processing system having a non-uniform memory architecture in which a cache adaptively prioritizes local or non-local data for replacement based on a cache replacement policy and a proportion of local data to non-local data stored at the cache, in accordance with some embodiments.

[0015] Figure 2 is a block diagram of a processing system including a plurality of processing units partitioned into a plurality of subsets of dies, in accordance with some embodiments.

[0016] Figure 3 is a block diagram illustrating prioritization of local and non-local data based on a cache replacement policy and a proportion of local data to non-local data stored at the cache, in accordance with some embodiments.

[0017] Figure 4 is a flow diagram illustrating a method for adaptively prioritizing local or non-local data for replacement based on a cache replacement policy and a proportion of local data to non-local data stored at the cache, in accordance with some embodiments. DETAILED DESCRIPTION

[0018] As the demand and amount of physical resources required to meet that demand for processors such as central processing units (CPUs), graphics processing units (GPUs), and other parallel processors increases, the size of processor dies becomes a limiting factor as larger dies are more difficult and expensive to manufacture. Accordingly, some processing systems partition the processors of the processing system into multiple dies (also referred to herein as chiplets), which can be configured to act as a single processor in some cases or collectively as multiple processors and interface with an application program in other cases. In various embodiments, and as used herein, the term “chiplet” refers to any device having the following characteristics: 1) composed of multiple active silicon dies each containing a portion of the computational logic used to solve a complete problem (i.e., the computational workload is distributed across multiple of these active silicon dies); 2) packaged together as a monolithic unit on the same substrate; and 3) the programming model can be unaware of the fact that the combination of these active silicon dies forms a single monolithic unit (i.e., each chiplet is not necessarily exposed to the application program as a separate device). Thus, in some implementations, a single processor is constructed using multiple chiplets that can be configured depending on the computational needs.

[0019] In processing systems implementing Non-Uniform Memory Architecture (NUMA), such as those with multiple dies, data stored in the cache can be accessed from local memory devices via local memory channels (referred to herein as "local data") and from non-local memory devices via non-local memory channels (referred to herein as "non-local data"). While local memory accesses can be performed relatively quickly, non-local memory accesses can take a relatively long time to complete. Therefore, memory management protocols that rely solely on the time when data was most recently requested, while ignoring the latency introduced by non-local memory accesses, can lead to lower performance and negatively impact the user experience. Furthermore, the impact of NUMA becomes more pronounced as processing systems are distributed across increasingly larger chiplets.

[0020] Figures 1 to 4 This illustrates a technique for a processing system to adjust cache replacement priorities for local and non-local data stored in the cache based on a cache replacement policy. In some implementations, the processing system comprises multiple chiplet subsets, and the cache is an intermediate cache located in a first chiplet subset. Local data requests originate from the local processor and arrive at the cache via a local memory channel. Non-local data requests originate from a non-local processor in a second (different) chiplet and arrive at the cache via a non-local memory channel. The cache controller selects the data to be replaced in the corresponding cache. This can be done in part based on the priorities assigned to local and non-local data stored in the cache. For example, in some implementations, the default behavior of the cache controller is to apply a Least Recently Used (LRU) replacement policy, where a cache line is selected for replacement based on the amount of time a cache line has been stored in the cache without being requested by an associated processor (referred to as the "age" of the data or cache line). In such implementations, the cache controller can adjust the replacement policy based on locality (i.e., whether a given cache line is local).

[0021] In some implementations, the cache controller periodically measures the amount of allocated local data (e.g., by counting the number of cache lines accessed via local memory channels) and non-local data (e.g., by counting the number of cache lines accessed via non-local memory channels). Based on the ratio of local to non-local data and the cache replacement policy, the cache controller can dynamically adjust the priority of local and non-local data in its replacement policy.

[0022] For example, in some implementations, the cache controller increases the priority of non-local data (i.e., increases the likelihood of replacement) in response to the amount of non-local data stored in the cache exceeding the amount of local data stored in the cache by more than a first threshold amount. In some implementations, the cache controller increases the priority of local data in response to the amount of local data stored in the cache exceeding the amount of non-local data stored in the cache by more than a second threshold amount. In some implementations, the second threshold amount is the same as the first threshold amount, while in other implementations, the second threshold amount is different from the first threshold amount. In some implementations, if the amount of non-local data stored in the cache does not exceed the first threshold amount of local data stored in the cache, and the amount of local data stored in the cache does not exceed the second threshold amount of non-local data stored in the cache, the cache controller assigns equal priorities to local and non-local data stored in the cache. In some implementations, the cache replacement policy (and therefore, the allocation of priorities) is based on the application executing at the processing system. By biasing the processing system's cache replacement policy based on the locality of the data stored in the cache, the processing system improves the cache replacement policy, thereby increasing processing efficiency.

[0023] Figure 1 A processing system 100 with a memory hierarchy according to some embodiments is illustrated, which employs a cache replacement strategy based at least in part on the locality of data residing in a cache that stores data accessed from both local and non-local memory channels. The processing system 100 includes a cache 102 at a local die 120, a non-local die 130, and memory 140. The local die 120 also includes a processor 108 and a cache controller 104. The local die 120 communicates with memory 110 attached to the local die 120 via a local memory channel 106, and with the non-local die 130 and memory 140 via a bus 118. The non-local die 130 includes a cache controller 136 and a cache 132. The bus 118 supports data flow between endpoints within the processing system 100. Some implementations of bus 118 are implemented as a Peripheral Component Interconnect (PCI) bus, a PCI-E bus, or other types of bus that support data flow between connection points such as the Peripheral Component Interface (PCI) physical layer, memory controller, Universal Serial Bus (USB) hub, computing and execution units including processor 108, and other endpoints. Components of processing system 100 may be implemented as hardware, firmware, software, or any combination thereof. It should be understood that processing system 100 may include, in addition to… Figure 1 Other than or related to those components shown in the diagram Figure 1The processing system 100 may include one or more software, hardware, and firmware components that differ from those shown in the diagram. For example, the processing system 100 may additionally include one or more input interfaces, non-volatile storage devices, one or more output interfaces, network interfaces, and one or more displays or display interfaces. The processing system 100 includes, for example, servers, desktop computers, laptop computers, tablet computers, mobile phones, game consoles, etc.

[0024] Processor 108 is implemented as a central processing unit (CPU), graphics processing unit (GPU), dedicated processor core or accelerator, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), etc. Processor 108 includes one or more instruction pipelines to execute instructions, thereby performing tasks on behalf of the electronic device. While processor 108 may have a certain amount of overall memory, for example in the form of a register file, such memory is typically limited in terms of storage capacity. Therefore, in order to execute instructions, processor 108 stores data and retrieves data from the memory hierarchy of processing system 100, which includes a cache hierarchy consisting of caches 102, 132, off-die memory 110 attached to local die 120, and memory 140. Specifically, during instruction execution, processor 108 generates operations called memory access requests to store (store operation) data at the memory hierarchy or load (load operation) data from the memory hierarchy. The cache hierarchy and memory 140 work together to satisfy memory access requests, as further described herein.

[0025] The memory controller 145 operates as an interface between the cache hierarchy and the memory 140. Therefore, data to be cached in the cache hierarchy is typically manipulated as blocks of data called "cache lines" and addressed or otherwise located within the memory hierarchy using the physical address of the memory 140. The memory controller 145 accesses cache lines from the memory 140 in response to memory requests from the cache hierarchy, and the cache lines are stored or cached in one or more caches 102, 132 of the cache hierarchy. Similarly, when a cache line containing modified data is evicted from the cache hierarchy and therefore needs to be updated in the memory 140, the memory controller 145 manages this write-back process.

[0026] Although only one cache is shown on each die in the example illustrated, a cache hierarchy may include one or more tiers of caches, such as a first-level (L0) cache, a second-level (L1) cache, and a third-level (L2) cache. In other embodiments, the cache hierarchy may include fewer or more than three tiers. The caches in the cache hierarchy are used to cache data for access and manipulation by the processor 108. Typically, caches at lower tiers (e.g., L0) tend to have lower storage capacity and lower access latency, while caches at higher tiers (e.g., L2) tend to have higher storage capacity and higher access latency. Therefore, cache lines of data are transferred between caches at different cache tiers to better optimize the utilization of cached data, given the storage capacity and access latency of the caches, through cache line eviction and cache line mounting processes managed by the cache logic of the cache controllers (e.g., cache controller 104) of the respective caches in the cache hierarchy.

[0027] Cache controller 104 receives a memory access request for data from processor 108 and searches cache 102 to determine if one of these cache entries stores a cache line associated with the memory address targeted by the memory access request. If the requested cache line is found in cache 102, a cache hit has occurred. In the case of a cache hit, cache controller 104 satisfies the memory access request by providing the requested cache line from cache 102 to processor 108 in the case of a read operation, or by storing write data into a cache entry in the case of a write operation. If the requested cache line is not found in cache 102, a cache miss has occurred. In the case of a cache miss at cache 102, cache controller 104 provides the memory access request to off-die memory 110 attached to local die 120 via local memory channel 106.

[0028] In some embodiments, off-die memory 110 is memory that shares an address space with memory 140. A memory controller (not shown) for off-die memory 110 receives a memory access request from cache controller 104 in response to a cache miss at cache 102. In response to receiving the memory access request, the memory controller identifies data to be retrieved from off-die memory 110 associated with the memory address targeted by the memory access request.

[0029] Similar to cache 102, cache 132 is a memory comprising a set of entries, each configured to store a cache line. Cache controller 136 receives a memory access request from cache controller 104. In response to receiving a memory access request, cache controller 136 identifies whether an entry in cache 132 stores data associated with the memory address targeted by the memory access request. In the case of a cache hit, cache controller 136 provides the requested data from cache 132 to cache 102, thus satisfying the memory access request as described above. In the case of a cache miss at cache 132, cache controller 136 provides the memory access request to memory 140. Memory controller 145 retrieves a cache line at the memory address targeted by the request in memory 140 and provides the cache line to cache 132, which in turn provides the cache line to cache 102, where the memory access request is satisfied. Thus, as described above, the memory access request traverses the memory hierarchy until the requested data is found. The requested data is then transferred to cache 102, where the memory access request is satisfied.

[0030] In some implementations, each of caches 102 and 132 is a set of associative caches, wherein each cache is divided into multiple sets. Each set includes multiple paths, each path corresponding to a cache entry storing a cache line. Each set stores only cache lines associated with a subset of memory addresses, wherein the subset associated with a set is identified by the corresponding cache controller based on a portion of the memory address called an index. By employing set associativity, caches 102 and 132 facilitate relatively fast identification of cache misses and cache hits.

[0031] In some implementations, the size of cache 102 is set such that it typically cannot store all the data requested or potentially requested by processor 108 at a given point in time, thus requiring data transfer via the memory hierarchy as described above. To ensure data consistency and efficient data transfer through the memory hierarchy, cache controller 104 implements a replacement policy to identify whether an entry exists in the set that is available to store a received cache line, and if not, selects one entry from the set for replacement. The availability of a cache entry is indicated by state information associated with that entry, referred to as the entry's validity status. Specifically, a cache line with an invalid status (referred to herein as an invalid cache line) is a cache line that is available to store data, and a cache line with a valid validity status (referred to herein as a valid cache line) is a cache line that is not available to store data unless replaced. To replace the valid cache line at an entry with an incoming cache line, the cache controller of the cache first evicts the valid cache line by transferring it to one or more other levels of the memory hierarchy, and then stores the incoming cache line at the entry.

[0032] For illustration, in response to a reset of processing system 100, cache controller 104 invalidates all cache lines in cache 102. When a cache entry is filled by a cache line retrieved from memory 140, cache controller 104 invalidates that cache entry. A cache path containing a cache line that has been invalidated can receive an incoming cache line that will replace or overwrite the invalid cache line. When cache 102 receives a cache line to store, it must select a cache path to store the cache line. If the cache set associated with the incoming cache line has available space (i.e., has one or more cache paths indicated to contain an invalid cache line), the incoming cache line will be stored at an invalid path. However, if all cache paths in the set associated with the incoming cache line are indicated to be valid, cache controller 104 selects a cache line in that set associated with the new cache line to evict it, thereby freeing up space for the incoming cache line.

[0033] The specific criteria used by cache controller 104 to select cache lines for replacement are called replacement policies. For example, in some embodiments, cache controller 104 implements cache replacement policy 112 at cache 102, where the least recently used cache line in the cache set associated with the incoming cache line (i.e., the cache line least recently targeted by memory access operations) is selected for eviction. In other embodiments, cache controller 104 implements cache replacement policy 112 at cache 102, where the least frequently used cache line in the cache set associated with the incoming cache line (i.e., the cache line historically the least frequently targeted by memory access operations within a specific time period) is selected for eviction. Other replacement policies include rereference interval prediction.

[0034] It will be understood that access via non-local memory channel 116 takes longer than access via local memory channel 106 because it must traverse a longer distance and may encounter latency at, for example, bus 118. The variable cache latency caused by the difference in cache miss times satisfied via local memory channels versus non-local memory channels negatively impacts processing performance. In some embodiments, cache controller 104 includes measurement circuitry 114. Measurement circuitry 114 periodically collects information about the ratio of local data to non-local data stored in cache 102. For example, in some embodiments, measurement circuitry 114 maintains a count of the number of cache lines residing in cache 102 retrieved via local memory channel 106 and a count of the number of cache lines residing in cache 102 retrieved via non-local memory channel 116.

[0035] To facilitate more efficient caching of both local and non-local data at cache 102, cache controller 104 applies cache replacement policy 112 based at least in part on the locality (i.e., local vs. non-local origin) of the data stored at cache 102. Depending on factors such as cache access history, the needs of the application (not shown) executing at processor 108, and the ratio of local to non-local data stored at cache 102, cache controller 104 prioritizes each cache line based on whether it is accessed via local memory channel 106 or non-local memory channel 116, as described in more detail below.

[0036] In some implementations, in such Figure 2 The processing system 100 is implemented in the multi-chip system shown. Figure 2This is a block diagram of a processing system 100 comprising multiple processing units divided into multiple die subsets according to some embodiments. According to some embodiments, the processing system 100 implements a multi-die processing unit 201. In various embodiments, the multi-die processing unit 201 is a parallel processor comprising any cooperative set of hardware and / or software that performs functions and computations associated with accelerated graphics processing tasks, data-parallel tasks, and nested data-parallel tasks in an accelerated manner relative to resources such as conventional CPUs, conventional graphics processing units (GPUs), and combinations thereof.

[0037] The processing system 200 includes one or more central processing units (CPUs) 250. Although Figure 1 A CPU 250 is shown, but some implementations of the processing system 100 include more CPUs. Bus 245 supports data flow between endpoints within the processing system 200, including the multi-die processing unit 201 and CPU 250, as well as other endpoints.

[0038] In various embodiments, CPU 250 is connected to memories 270, 272, and 274 via bus 245, which in some embodiments are implemented as dynamic random access memory (DRAM). In various embodiments, memories 270, 272, and 274 may also be implemented using other types of memory, including static random access memory (SRAM), non-volatile RAM, etc. In the illustrated embodiments, CPU 250 communicates with memories 270, 272, and 274 and the multi-die processing unit 201 via bus 245. However, some embodiments of the processing system 200 include a multi-die processing unit 201 that communicates with CPU 250 via direct connection or via a dedicated bus, bridge, switch, router, etc.

[0039] As shown in the figure, CPU 250 includes multiple processes, such as executing one or more application programs 260 to generate graphics commands. In various embodiments, one or more application programs 260 include applications that utilize the functionality of the multi-chip processing unit 201, such as applications that generate work within the processing system 200 or operating system (OS). In some specific embodiments, application program 260 includes one or more graphics instructions instructing the multi-chip processing unit 201 to render a graphical user interface (GUI) and / or a graphics scene. For example, in some specific embodiments, graphics instructions include instructions that define a set of one or more graphics primitives to be rendered by the multi-chip processing unit 201.

[0040] In some implementations, application 260 utilizes a graphics application programming interface (API) 265 to invoke a user-mode driver (not shown) (or a similar GPU driver). The user-mode driver issues one or more commands to the multi-die processing unit 201 for rendering one or more graphics primitives into a displayable graphical image. Based on the graphics instructions issued by application 260 to the user-mode driver, the user-mode driver formulates one or more graphics commands that specify one or more operations performed by the multi-die graphics primitive 201 for rendering graphics. In some implementations, the user-mode driver is part of application 260 running on CPU 250. For example, in some implementations, the user-mode driver is part of a game application running on CPU 250. Similarly, in some implementations, a kernel-mode driver (not shown) formulates the one or more graphics commands, either alone or in combination with the user-mode driver, as part of an operating system running on CPU 250.

[0041] Figure 2 The multi-die processing unit 201 shown includes three chipsets 202, 203, and 204. Each chipset 202, 203, and 204 includes a set of shader engine (SE) dies 205 for concurrently or in parallel receiving and executing commands. In some embodiments, each SE die 205 includes a configurable number of shader engines, wherein each shader engine includes a configurable number of workgroup processors, and each workgroup processor includes a configurable number of compute units. Some embodiments of the SE dies 205 use information from draw calls received from the CPU 250 to configure shading of the vertices of primitives representing a model of the scene. Although three SE dies 205 are illustrated for each GPU chipset 202, 203, and 204, this makes it possible to... Figure 2 The diagram shows a total of nine SE dies 205, but some implementations of the multi-die processing unit 201 include more or fewer shader engine dies 205. Furthermore, although... Figure 2 The figure shows three L2 cache dies 210, 220, and 230, but the number of L2 cache dies and corresponding shader engine dies in the multi-die processing unit 201 is a design choice issue and varies in other implementations.

[0042] Each SE die 205 includes an L1 cache 208 to store data directly accessed by the corresponding SE die 205. Additionally, each set of SE dies 205 in chipsets 202, 203, 204 is connected to an L2 cache die (e.g., L2 cache dies 210, 220, and 230), which includes corresponding L2 caches 212, 222, 232 storing frequently used data and instructions, including both local and non-local data, for access by the shader engine and corresponding L2 cache controllers 214, 224, 234 at each SE die 205. In some embodiments, each L2 cache 212, 222, 232 is connected to one or more L1 caches 208 implemented in the SE die 205 and one or more L3 caches (or other last-level caches) implemented in the processing system 200. The caches together form a cache hierarchy. Each L2 cache die 210, 220, 230 is also connected to the corresponding off-die memory 270, 272, 274 via the corresponding local memory channels 275, 226.

[0043] exist Figure 2 In the depicted implementation, the consistent data texture 240 communicatively couples L2 cache dies 210, 220, and 230 to each other. The consistent data texture 240 includes levers for routing memory access requests and responses between any memory accesses. In some implementations, the consistent data texture 240 also includes a system memory map defined by the Basic Input / Output System (BIOS) for determining the destination of memory accesses based on system configuration, and buffers for each virtual connection.

[0044] As a general operational overview, L2 cache 212 is communicatively coupled to each L1 cache in the L1 cache 208 of the corresponding SE die 205 of chiplet 202 via data bus 206, and L2 cache die 210 is communicatively coupled to memory 270 via local memory channel 275. Similarly, L2 cache 222 is communicatively coupled to each L1 cache in the L1 cache 208 of the corresponding SE die 205 of chiplet 203 via a corresponding data bus, and L2 cache 232 is communicatively coupled to each L1 cache in the L1 cache 208 of the corresponding SE die 205 of chiplet 204 via a corresponding data bus, and L2 cache die 220 is communicatively coupled to memory 272 via local memory channel 226. L2 cache 212 is communicatively coupled to each L1 cache in L1 cache 208 of SE die 205 of chipsets 203, 204 via non-local memory channels 216 of the access coherent data texture 240. Similarly, L2 caches 222, 232 are communicatively coupled to L1 cache 208 of SE die 205 of chipset 203 (and to another chipset in chipsets 202, 204) via corresponding non-local memory channels. Thus, any inter-chipset memory access is appropriately routed via the coherent data texture 240 to access memory channels on other chipsets. In this way, the multi-die processing unit 201 includes GPU chipsets that can be addressed as a single, monolithic processing unit from the perspective of a software developer (e.g., the CPU 250 and any associated applications / drivers are unaware of the chipet-based architecture), and thus avoids the need for programmers or developers to make any chipet-specific considerations.

[0045] However, any chiplet-to-chipset memory access incurs latency due to the need to traverse non-local memory channels. For example, in response to a cache miss for non-local data at L2 cache 212 and memory 270 connected to L2 cache die 210 via local memory channel 275, L2 cache controller 214 provides a memory access request to L2 cache controller 224 at chiplet 203 via consistent data texture 240. L2 cache controller 224 receives the memory access request from L2 cache controller 214. In response to receiving the memory access request, L2 cache controller 224 identifies whether an entry in L2 cache 222 stores data associated with the memory address targeted by the memory access request. In the case of a cache hit, L2 cache controller 224 provides the requested data from L2 cache 222 to L2 cache 212 via consistent data texture 240, thus satisfying the memory access request. In the event of a cache miss at L2 cache 222, L2 cache controller 224 provides a memory access request to memory 272 via memory channel 226. This request is non-local to the L2 cache die 210 from which the cache miss originated. The memory controller (not shown) retrieves a cache line at the memory address in memory 272 targeted by the request and provides the cache line to L2 cache 232, which in turn provides a cache line to L2 cache 212, where the memory access request is satisfied.

[0046] Figure 3 This is a block diagram illustrating how local and non-local data are prioritized based on a cache replacement policy and the ratio of local to non-local data stored in the cache, according to some implementation schemes. If a non-random replacement policy is used, the cache effectively maintains an ordered list of eviction candidates. In the case of LRU or Most Recently Used (MRU) type policies, this ordered list can be referred to as a reference chain of entries, where accessed entries are promoted to the head of the chain according to the replacement policy. Under the LRU policy, entries are evicted from the tail of the chain, while under the MRU policy, entries are evicted from the head of the chain.

[0047] In the example shown, L2 cache 212 includes a reference chain of cache entries. Under a cache replacement policy that does not support either local or non-local data, data fetched into L2 cache 212 is initially placed in the middle of the reference chain. If data is subsequently requested, the data moves towards the head 301 of the reference chain. When additional data is fetched into L2 cache 212, data initially stored in the middle of the reference chain, if not requested, moves towards the bottom of the reference chain, referred to as the tail 302. When data is evicted from L2 cache 212, it is replaced from the tail 302 of the reference chain. Under cache replacement policy 300, L2 cache controller 214 assigns all data 322 (including both local and non-local data) to the same priority within the reference chain when they are first fetched into L2 cache 212.

[0048] Under cache replacement policy 310, non-local data 324 is prioritized to reduce average latency. Based on cache replacement policy 310, L2 cache controller 214 assigns a priority to local data 326, placing it in the middle of the reference chain when it is first retrieved, and assigns a priority to non-local data 324, placing it towards the head 301 of the reference chain. Therefore, local data 326 is more likely to arrive at the tail 302 of the reference chain before non-local data 324, and thus faces a higher probability of being evicted from L2 cache 212 compared to non-local data 324.

[0049] Conversely, under cache replacement policy 320, local data 322 is prioritized. For example, in the case of a non-uniform miss rate between local data 322 and non-local data 324, the placement of local data can be prioritized. To illustrate, if the rereference interval for non-local data 324 makes it always avoid the cache, but the shorter rereference interval for local data 322 keeps it in the cache, then cache replacement policy 320 can prioritize local data 322. Based on cache replacement policy 320, L2 cache controller 214 assigns a priority to non-local data 324, placing it in the middle of the reference chain when it is first retrieved, and assigns a priority to local data 326, placing it towards the head 301 of the reference chain. Therefore, non-local data 324 is more likely to arrive at the tail 302 of the reference chain before local data 326, and thus faces a higher probability of being evicted from L2 cache 212 compared to local data 326. Therefore, the priority assigned to local and non-local data is represented by the initial proximity of the data to the head 301 or tail 302 of the L2 cache 212.

[0050] In some implementations, the logic at the L2 cache controller 214 applies a cache replacement policy that prioritizes local or non-local data based on the needs of the application 260 executing at the processing system 200. In some implementations, the L2 cache controller 214 adjusts the corresponding priorities of local and non-local data based on the relative proportions of local and non-local data stored in the L2 cache 212.

[0051] Figure 4 This is a flowchart illustrating a method 400 for adaptively prioritizing local or non-local data based on a cache replacement strategy and the ratio of local to non-local data stored in the cache, according to some implementations. In some implementations, method 400 is used in, for example... Figure 1 Processing system 100 or Figure 2 The processing system 200 is implemented in the processing system.

[0052] At block 402, the L2 cache controller 214 applies a cache replacement policy that does not support either local or non-local data. Therefore, at block 402, the L2 cache controller 214 assigns equal priority to both local data 326 and non-local data 324. In some implementations, the L2 cache controller 214 therefore places all data 322 in the middle of the reference chain of the L2 cache 212.

[0053] At block 404, the measurement circuitry 114 of the L2 cache controller 214 periodically collects information about the amount of local data 326 and non-local data 324 stored in the L2 cache 212 (e.g., the number of cache lines). Based on the information collected by the measurement circuitry 114, the L2 cache controller 214 determines the relative proportions of local data 326 and non-local data 324 stored in the L2 cache 212.

[0054] In the example shown, at block 406, the L2 cache controller 214 determines whether the amount of non-local data 324 stored in L2 cache 212 exceeds the amount of local data 326 stored in L2 cache 212 by more than a first threshold amount. In other embodiments, the L2 cache controller 214 determines whether the ratio of non-local data 324 to local data 326 stored in L2 cache 212 exceeds a first threshold amount. If at block 406, the L2 cache controller 214 determines that the first threshold has been exceeded, the method flow continues to block 410.

[0055] At block 410, the L2 cache controller 214 prioritizes the non-local data 324 for replacement. In some implementations, the L2 cache controller 214 prioritizes the non-local data 324 by assigning it a higher replacement priority. For example, the L2 cache controller 214 assigns a priority to local data 326 to place it toward the head 301 of the reference chain of the L2 cache 212, and assigns a priority to place the non-local data 324 in the middle (or closer to the tail 302) of the reference chain. The method flow then continues back to block 404.

[0056] If at block 406, the L2 cache controller 214 determines that a first threshold has not been exceeded, the method flow continues to block 408. At block 408, the L2 cache controller 214 determines whether the amount of local data 326 stored in L2 cache 212 exceeds the amount of non-local data 324 stored in L2 cache 212 by more than a second threshold amount. In other embodiments, the L2 cache controller 214 determines whether the ratio of local data 326 to non-local data 324 stored in L2 cache 212 exceeds a second threshold amount. In some embodiments, the first threshold and the second threshold are the same, and in other embodiments, the first threshold and the second threshold are different from each other. If at block 408, the L2 cache controller 214 determines that the second threshold has not been exceeded, the method flow continues back to block 402.

[0057] If, at box 408, the L2 cache controller 214 determines that the second threshold has been exceeded, the method flow continues to box 412. At box 412, the L2 cache controller 214 prioritizes the local data 326 for replacement. In some implementations, the L2 cache controller 214 prioritizes the local data 326 by assigning it a higher replacement priority. For example, the L2 cache controller 214 assigns a priority to non-local data 324 to place it towards the head 301 of the reference chain of the L2 cache 212, and assigns a priority to place the local data 326 in the middle of the reference chain (or closer to the tail 302). The method flow then continues back to box 404.

[0058] In some implementations, the above-described devices and techniques are implemented in systems including one or more integrated circuit (IC) devices (also known as integrated circuit packages or microchips), such as those referenced above. Figures 1 to 4The described processing system. Electronic design automation (EDA) and computer-aided design (CAD) software tools can be used in the design and manufacture of these IC devices. These design tools are typically represented as one or more software programs. One or more software programs include code executable by a computer system to manipulate the computer system to operate on code representing a circuit of one or more IC devices to perform at least a portion of a process for designing or adapting a manufacturing system to manufacture the circuit. The code may include instructions, data, or a combination of instructions and data. Software instructions representing design or manufacturing tools are typically stored in a computer-readable storage medium accessible to the computing system. Similarly, code representing one or more stages of the design or manufacture of an IC device may be stored in or accessed from the same computer-readable storage medium or different computer-readable storage media.

[0059] Computer-readable storage media can include any non-transitory storage medium or a combination of non-transitory storage media that can be accessed by a computer system during use to provide instructions and / or data to the computer system. Such storage media can include, but are not limited to, optical media (e.g., optical discs (CDs), digital versatile optical discs (DVDs), Blu-ray discs), magnetic media (e.g., floppy disks, magnetic tapes, or magnetic hard disks), volatile memory (e.g., random access memory (RAM) or cache), non-volatile memory (e.g., read-only memory (ROM) or flash memory), or microelectromechanical systems (MEMS) based storage media. Computer-readable storage media can be embedded in a computing system (e.g., system RAM or ROM), fixedly attached to a computing system (e.g., a magnetic hard disk drive), removably attached to a computing system (e.g., an optical disc or a flash memory based on a universal serial bus (USB)), or coupled to a computer system via a wired or wireless network (e.g., a network accessible storage device (NAS)).

[0060] In some implementations, certain aspects of the above-described techniques may be implemented by one or more processors of a processing system executing the software. The software includes one or more sets of executable instructions stored or otherwise tangibly embodied on a non-transitory computer-readable storage medium. The software may include instructions and certain data that, when executed by one or more processors, manipulate one or more processors to perform one or more aspects of the above-described techniques. The non-transitory computer-readable storage medium may include, for example, disk or optical disc storage devices, solid-state storage devices such as flash memory, cache, random access memory (RAM), or one or more other non-volatile memory devices. The executable instructions stored on the non-transitory computer-readable storage medium may be source code, assembly language code, object code, or other instruction formats that are interpreted or otherwise executed by one or more processors.

[0061] It should be noted that not all activities or elements described above in the general description are essential. A particular activity or part of the apparatus may not be essential, and one or more additional activities may be performed, or elements may be included in addition to those described. Furthermore, the order in which the activities are listed is not necessarily the order in which they are performed. Additionally, these concepts have been described with reference to specific embodiments. However, those skilled in the art will understand that various modifications and changes can be made without departing from the scope of this disclosure as set forth in the following claims. Therefore, the specification and drawings are to be considered illustrative rather than restrictive, and all such modifications are intended to be included within the scope of this disclosure.

[0062] The benefits, other advantages, and solutions to problems have been described above with respect to specific embodiments. However, the benefits, advantages, solutions to problems, and any features that may lead to or make any benefit, advantage, or solution appear or become more significant should not be construed as key, essential, or fundamental features of any or all claims. Furthermore, the specific embodiments disclosed above are merely illustrative, as the disclosed subject matter can be modified and practiced in different but equivalent ways that will be apparent to those skilled in the art who benefit from the teachings herein. No limitation is intended on the details of the constructions or designs shown herein, except as described in the following claims. Therefore, it will be apparent that the specific embodiments disclosed above can be changed or modified, and all such changes are considered to be within the scope of the disclosed subject matter. Therefore, the protection sought herein is set forth in the following claims.

Claims

1. A method comprising: allocating a priority for local data and a priority for non-local data based on a cache replacement policy, the local data stored at a cache at a first sub-set of small chips of a processing system comprising a plurality of sub-sets of small chips, wherein the local data comprises data accessed from local memory at the first sub-set of small chips via a local memory channel, the non-local data stored at the cache, wherein the non-local data comprises data accessed from non-local memory at a second sub-set of small chips of the processing system via a non-local memory channel; and selecting data for replacement at the cache based on the allocated priorities.

2. The method of claim 1, further comprising: periodically measuring an amount of local data stored at the cache and an amount of non-local data stored at the cache.

3. The method of claim 1 or claim 2, further comprising: adjusting the priority of local data stored at the cache and the priority of non-local data stored at the cache based on a ratio of local data to non-local data stored at the cache.

4. The method of any one of claims 1-3, further comprising: increasing the priority of non-local data in response to an amount of non-local data stored at the cache exceeding an amount of local data stored at the cache by more than a first threshold amount.

5. The method of any one of claims 1-3, further comprising: increasing the priority of local data in response to an amount of local data stored at the cache exceeding an amount of non-local data stored at the cache by more than a second threshold amount.

6. The method of any one of claims 1-3, further comprising: allocating equal priorities for local data and non-local data stored at the cache in response to an amount of non-local data stored at the cache not exceeding an amount of local data stored at the cache by a first threshold amount and the amount of local data stored at the cache not exceeding the amount of non-local data stored at the cache by more than a second threshold amount.

7. The method of any one of claims 1-5, wherein allocating is based on an application program executing at the processing system.

8. A processing system comprising: a cache at a first sub-set of dies of the processing system, the cache configured to store local data accessed from local memory at the first sub-set of dies via a local memory channel and non-local data accessed from non-local memory at a second sub-set of dies of the processing system via a non-local memory channel; and a cache controller configured to: allocate a priority for the local data and a priority for the non-local data based on a cache replacement policy; and select data for replacement at the cache based on the allocated priorities. ​ ​ selecting data for replacement at the cache based on the assigned priority.

9. The processing system of claim 8, wherein the cache controller is further to: periodically measure an amount of local data stored at the cache and an amount of non-local data stored at the cache.

10. The processing system of claim 8 or claim 9, wherein the cache controller is further to: adjust the priority of local data stored at the cache and the priority of non-local data stored at the cache based on a ratio of local data to non-local data stored at the cache.

11. The processing system of any of claims 8 to 10, wherein the cache controller is further to: increase the priority of non-local data in response to an amount of non-local data stored at the cache exceeding an amount of local data stored at the cache by more than a first threshold amount.

12. The processing system of any of claims 8 to 10, wherein the cache controller is further to: increase the priority of local data in response to an amount of local data stored at the cache exceeding an amount of non-local data stored at the cache by more than a second threshold amount.

13. The processing system of claim 8 or claim 9, wherein the cache controller is further to: assign equal priority to local data and non-local data stored at the cache in response to an amount of non-local data stored at the cache not exceeding an amount of local data stored at the cache by a first threshold amount and the amount of local data stored at the cache not exceeding the amount of non-local data stored at the cache by more than a second threshold amount.

14. The processing system of any of claims 8 to 12, wherein the cache controller is further to: assign priority based on an application program executing at the processing system.

15. A processing system, the processing system comprising: a cache at a first subset of dies of the processing system; and a cache controller configured to: adaptively prioritize one of local data and non-local data for replacement at the cache based on a cache replacement policy and a ratio of local data to non-local data stored at the cache, wherein: local data comprises data accessed from local memory at the first subset of dies via a local memory channel; and non-local data comprises data accessed from non-local memory at a second subset of dies of the processing system via a non-local memory channel.

16. The processing system of claim 15, wherein the cache controller is further to: periodically measure an amount of local data stored at the cache and an amount of non-local data stored at the cache.

17. The processing system of claim 15 or claim 16, wherein the cache controller is further to: increase the priority of non-local data in response to an amount of non-local data stored at the cache exceeding an amount of local data stored at the cache by more than a first threshold amount.

18. The processing system of claim 15 or claim 16, wherein the cache controller is further to: increase the priority of local data in response to an amount of local data stored at the cache exceeding an amount of non-local data stored at the cache by more than a second threshold amount.

19. The processing system of claim 15 or claim 16, wherein the cache controller is further to: assign equal priority to local data and non-local data stored at the cache in response to an amount of non-local data stored at the cache not exceeding an amount of local data stored at the cache by a first threshold amount and the amount of local data stored at the cache not exceeding the amount of non-local data stored at the cache by more than a second threshold amount.

20. The processing system of any one of claims 15 to 18, wherein the cache controller is further to: assign priority based on an application program executing at the processing system.

Images