Intermediate cache management for non-uniform memory architectures

Abstract

A cache controller (104) of a processing system (100) implementing a non-uniform memory architecture (NUMA) adjusts the cache replacement priority of local and non-local data stored in the cache based on a cache replacement policy (112). Local data (326) is data accessed by the cache (102) via a local memory channel (106), and non-local data (324) is data accessed by the cache (102) via a non-local memory channel (116). Based on the cache replacement policy, the cache controller assigns priorities to the local and non-local data stored in the cache and selects data to be replaced in the cache, at least partially based on the assigned priorities.

Description

Background Art

[0001] A processing system typically includes a memory subsystem having a memory module for storing data accessed by executed instructions. The memory subsystem has a main memory at the top of the hierarchy for storing a larger amount of data that can be accessed by the executing instructions, and is organized into a memory hierarchy having one or more caches at lower levels of the memory hierarchy for storing a subset of the data stored in the main memory. Typically, the lower the data stored in the memory hierarchy, the more quickly the processor can access it. To further improve processing efficiency, the processing system can implement a memory management protocol for managing a specific data set stored at each level of the memory hierarchy. For example, the processing system can implement a memory management protocol that moves recently requested data for access to a lower level of the memory hierarchy expecting the data to be accessed again by a processor core in the near future, and moves data that has not been recently accessed to a higher level of the memory hierarchy.

Summary of the Invention

Means for Solving the Problems

[0002] The examples described herein illustrate techniques for a processing system to adjust cache replacement priorities for local and non-local data stored in a cache based on a cache replacement policy. In some embodiments, the processing system includes a subset of multiple chiplets, and the cache is an intermediate cache located in a subset of first chiplets. In the first example, the method includes assigning, based on a cache replacement policy, a priority for local data stored in a cache in a subset of first chiplets of a processing system including a subset of multiple chiplets, where local data includes data accessed from local memory in the subset of first chiplets via a local memory channel, and a priority for non-local data stored in the cache, where non-local data includes data accessed from non-local memory in a subset of second chiplets of the processing system via a non-local memory channel. The method further includes selecting data to be replaced in the cache based on the assigned priorities.

[0003] In some embodiments, the method further includes periodically measuring the amount of local data stored in the cache and the amount of non-local data stored in the cache. The method may also include adjusting the priority of local data stored in the cache and the priority of non-local data stored in the cache based on the ratio of local data to non-local data stored in the cache.

[0004] In some embodiments, the method further includes prioritizing non-local data in response to the amount of non-local data stored in the cache exceeding a first threshold amount by a certain amount compared to the amount of local data stored in the cache. In other embodiments, the method further includes prioritizing local data in response to the amount of local data stored in the cache exceeding a second threshold amount compared to the amount of non-local data stored in the cache.

[0005] The method may further include assigning equal priorities to local and non-local data stored in the cache, depending on whether the amount of non-local data stored in the cache exceeds the amount of local data stored in the cache by a first threshold amount, and whether the amount of local data stored in the cache exceeds the amount of non-local data stored in the cache by a second threshold amount. In some embodiments, the assignment of priorities to local and non-local data is based on the application being run in the processing system.

[0006] In another example, the processing system includes a cache located on a subset of the first die of the processing system, configured to store local data accessed from local memory on a subset of the first die via a local memory channel, and non-local data accessed from non-local memory on a subset of the second die of the processing system via a non-local memory channel. The processing system further includes a cache controller configured to assign priorities to local data and non-local data based on a cache replacement policy, and to select data for replacement in the cache based on the assigned priorities.

[0007] In some embodiments, the cache controller periodically measures the amount of local data and non-local data stored in the cache. The cache controller may also adjust the priority of local data and non-local data stored in the cache based on the ratio of local data to non-local data stored in the cache.

[0008] For example, in some embodiments, the cache controller is configured to prioritize non-local data when the amount of non-local data stored in the cache exceeds the amount of local data stored in the cache by a first threshold amount. In other embodiments, the cache controller is configured to prioritize local data when the amount of local data stored in the cache exceeds the amount of non-local data stored in the cache by a second threshold amount.

[0009] The cache controller may be configured to assign equal priority to local and non-local data stored in the cache, depending on whether the amount of non-local data stored in the cache exceeds the amount of local data stored in the cache by a first threshold amount, and whether the amount of local data stored in the cache exceeds the amount of non-local data stored in the cache by a second threshold amount. In some embodiments, the cache controller is further configured to assign priority to local and non-local data, respectively, based on the application being executed in the processing system.

[0010] In a further example, the processing system includes a cache located on a subset of the first die of the processing system, and a cache controller, the cache controller being configured to adaptively prioritize either local or non-local data for replacement in the cache based on a cache replacement policy and the ratio of local data to non-local data stored in the cache, where local data includes data accessed from local memory located on a subset of the first die via a local memory channel, and non-local data includes data accessed from non-local memory located on a subset of the second die of the processing system via a non-local memory channel.

[0011] The cache controller may be further configured to periodically measure the amount of local data stored in the cache and the amount of non-local data stored in the cache. In some embodiments, the cache controller is further configured to prioritize non-local data when the amount of non-local data stored in the cache exceeds the amount of local data stored in the cache by a first threshold amount. In other embodiments, the cache controller is further configured to prioritize local data when the amount of local data stored in the cache exceeds the amount of non-local data stored in the cache by a second threshold amount.

[0012] The cache controller may be configured to assign equal priority to local and non-local data stored in the cache, depending on whether the amount of non-local data stored in the cache exceeds the amount of local data stored in the cache by a first threshold amount, and whether the amount of local data stored in the cache exceeds the amount of non-local data stored in the cache by a second threshold amount. Furthermore, in some embodiments, the cache controller is further configured to assign priority to local and non-local data, respectively, based on the application being executed in the processing system.

[0013] This disclosure will be better understood by referring to the accompanying drawings, and many of its features and advantages may become apparent to those skilled in the art. The use of the same reference numerals in different drawings indicates similar or identical items. [Brief explanation of the drawing]

[0014] [Figure 1]This is a block diagram of a processing system having a non-uniform memory architecture in which the cache adaptively prioritizes local or non-local data to be replaced based on a cache replacement policy and the ratio of local data to non-local data stored in the cache, according to several embodiments. [Figure 2] This is a block diagram of a processing system including multiple processing units partitioned into subsets of multiple dies, according to several embodiments. [Figure 3] This block diagram shows the prioritization of local and non-local data based on a cache replacement policy and the ratio of local data to non-local data stored in the cache, according to several embodiments. [Figure 4] This flowchart illustrates a method for adaptively prioritizing local or non-local data based on a cache replacement policy and the ratio of local data to non-local data stored in the cache, according to several embodiments. [Modes for carrying out the invention]

[0015] As the demand for processors such as central processing units (CPUs), graphics processing units (GPUs), and other parallel processors increases, and as the amount of physical resources required to meet those demands increases, the size of the processor die becomes a limiting factor because larger dies are more difficult and expensive to manufacture. Therefore, some processing systems divide the processor of the processing system into multiple dies (also referred to herein as chiplets) that can function collectively and interface with applications, either as a single processor in some examples or as multiple processors in other examples. In various embodiments, as used herein, the term "chiplet" refers to any device having the following characteristics: 1) several active silicon dies, each containing a portion of the computational logic used to solve a complete problem (i.e., the computational workload is distributed across multiple of these active silicon dies); 2) some of these are packaged together as a monolithic unit on the same substrate; and 3) the programming model can be independent of the fact that these combinations form a single monolithic unit (i.e., each chiplet is not necessarily exposed to the application as a separate device). Therefore, in some embodiments, a single processor is constructed using several chiplets that can be configured according to the computational needs.

[0016] In processing systems that implement a non-uniform memory architecture (NUMA), such as those with multiple dies, data stored in the cache can be accessed both from local memory devices via local memory channels (referred to herein as "local data") and from non-local memory devices via non-local memory channels (referred to herein as "non-local data"). Local memory access can be performed relatively quickly, while non-local memory access can take a relatively long time to complete. Therefore, a memory management protocol that relies solely on how recently the data was requested, while overlooking the latency introduced by non-local memory access, can result in lower performance and negatively impact the user experience. Furthermore, the effects of NUMA become more severe as processing systems are distributed across an increasing number of chiplets.

[0017] Figures 1-4 illustrate techniques for a processing system to adjust the cache replacement priority of local and non-local data stored in the cache based on a cache replacement policy. In some embodiments, the processing system includes a subset of multiple chiplets, and the cache is an intermediate cache located in a subset of the first chiplets. Local data requests originate from a local processor and arrive in the cache via a local memory channel. Non-local data requests originate from a non-local processor in a second (different) chiplet and arrive in the cache via a non-local memory channel. The cache controller selects the data to be replaced in the corresponding cache. This can be done in part based on the priority assigned to local and non-local data stored in the cache. For example, in some embodiments, the default behavior of the cache controller is to apply a Longest Unused (LRU) replacement policy, which selects cache lines for replacement based on the amount of time (referred to as the "elapsed time" of the data or cache line) that has been stored in the cache without being requested by the relevant processor. In such embodiments, the cache controller can adjust the replacement policy based on locality (i.e., whether a given cache line is local or not).

[0018] In some embodiments, the cache controller periodically measures the amount of allocated local data (e.g., by counting the number of cache lines accessed via local memory channels) and the amount of non-local data (e.g., by counting the number of cache lines accessed via non-local memory channels). Based on the ratio of local to non-local data and the cache replacement policy, the cache controller can dynamically adjust the priority of local and non-local data in its replacement policy.

[0019] For example, in some embodiments, the cache controller prioritizes non-local data (i.e., increases the likelihood of substitution) in response to the amount of non-local data stored in the cache exceeding the amount of local data stored in the cache by a first threshold amount. In some embodiments, the cache controller is configured to prioritize local data in response to the amount of local data stored in the cache exceeding the amount of non-local data stored in the cache by a second threshold amount. The second threshold amount is the same as the first threshold amount in some embodiments and different in other embodiments. In some embodiments, if the amount of non-local data stored in the cache does not exceed the amount of local data stored in the cache by the first threshold amount, and the amount of local data stored in the cache does not exceed the amount of non-local data stored in the cache by the second threshold amount, the cache controller assigns equal priority to the local and non-local data stored in the cache. In some embodiments, the cache substitution policy (and therefore the assignment of priority) is based on the application running on the processing system. By biasing the processing system's cache substitution policy based on the locality of the data stored in the cache, the processing system can improve the cache substitution policy, resulting in improved processing efficiency.

[0020] Figure 1 shows a processing system 100 having a memory hierarchy that employs a cache replacement policy based at least in part on the locality of data residing in a cache that stores data accessed from both local and non-local memory channels, according to several embodiments. The processing system 100 includes a cache 102 located on a local die 120, a non-local die 130, and memory 140. The local die 120 also includes a processor 108 and a cache controller 104. The local die 120 communicates with memory 110 attached to the local die 120 via a local memory channel 106, and communicates with the non-local die 130 and memory 140 via a bus 118. The non-local die 130 includes a cache controller 136 and a cache 132. The bus 118 supports data flow between endpoints within the processing system 100. Some embodiments of bus 118 may be implemented as a peripheral component interface (PCI) bus, a PCI-E bus, a peripheral component interface (PCI) physical layer, a memory controller, a universal serial bus (USB) hub, a computing and execution unit including processor 108, and other types of buses supporting data flow between connection points such as other endpoints. The components of processing system 100 may be implemented as hardware, firmware, software, or any combination thereof. It should be understood that processing system 100 may include one or more software components, hardware components, and firmware components in addition to or different from those shown in Figure 1. For example, processing system 100 may further include one or more input interfaces, non-volatile storage, one or more output interfaces, a network interface, and one or more displays or display interfaces. Processing system 100 includes, for example, servers, desktop computers, laptop computers, tablet computers, mobile phones, game consoles, etc.

[0021] The processor 108 is implemented as a central processor core (CPU), graphics processor unit core (GPU), dedicated processor core or accelerator, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), etc. The processor 108 includes one or more instruction pipelines for executing instructions, thereby performing tasks on behalf of electronic devices. The processor 108 may have some amount of internal memory, for example in the form of a register file, but such memory is typically limited in storage capacity. Therefore, in order to execute instructions, the processor 108 stores and retrieves data from the memory hierarchy of the processing system 100, which includes a cache hierarchy consisting of caches 102, 132, off-die memory 110 attached to the local die 120, and memory 140. In particular, in the process of executing instructions, the processor 108 generates operations called memory access requests to store data in the memory hierarchy (storage operations) or load data from the memory hierarchy (load operations). The cache hierarchy and memory 140 work together to satisfy memory access requests, as will be further described herein.

[0022] The memory controller 145 acts as an interface between the cache hierarchy and memory 140. Therefore, data cached in the cache hierarchy is typically operated as blocks of data referred to as "cache lines," which are addressed to or otherwise located in the memory hierarchy using the physical addresses of memory 140. Cache lines are accessed from system memory 140 by the memory controller 145 in response to access requests from processing clients, and the cache lines are stored or cached in one or more caches 102, 132 of the cache hierarchy. Similarly, if a cache line containing modified data needs to be retrieved from the cache hierarchy and therefore updated in memory 140, the memory controller 145 manages this write-back process.

[0023] In the illustrated example, only one cache is shown on each die, but the cache hierarchy includes one or more levels of caches, such as a first level (L0) cache, a second level (L1) cache, and a third level (L2) cache. In other embodiments, the cache hierarchy includes fewer or more levels than three. The caches in the cache hierarchy are used to cache data for access and manipulation by the processor 108. Typically, lower level (e.g., L0) caches tend to have lower storage capacity and lower access latency, while higher level (e.g., L2) caches tend to have higher storage capacity and higher access latency. Therefore, cache lines of data are transferred between caches of different cache levels to better optimize the use of cache data in terms of cache storage capacity and access latency, through cache line eviction and cache line installation processes managed by the cache logic of the cache controller (e.g., cache controller 104) of the individual caches in the cache hierarchy.

[0024] The cache controller 104 receives a memory access request for data from the processor 108, searches the cache 102, and determines whether any of the cache entries store a cache line associated with the memory address targeted by the memory access request. If the requested cache line is found within the cache 102, a cache hit has occurred. In the case of a cache hit, the cache controller 104 satisfies the memory access request by providing the requested cache line from the cache 102 to the processor 108 in the case of a read operation, or by storing the write data in the cache entry in the case of a write operation. If the requested cache line is not found within the cache 102, a cache miss has occurred. When a cache miss occurs in the cache 102, the cache controller 104 provides the memory access request to the off-die memory 110 attached to the local die 120 via the local memory channel 106.

[0025] The off-die memory 110 is, in some embodiments, a memory that shares an address space with the memory 140. A memory controller (not shown) for the off-die memory 110 receives the memory access request from the cache controller 104 in response to a cache miss in the cache 102. In response to receiving the memory access request, the memory controller identifies and retrieves data from the off-die memory 110 associated with the memory address targeted by the memory access request.

[0026] Similar to cache 102, cache 132 is a memory that includes a set of entries, and each entry is configured to store a cache line. Cache controller 136 receives a memory access request from cache controller 104. In response to receiving the memory access request, cache controller 136 identifies whether the entry of cache 132 stores data associated with the memory address targeted by the memory access request. In the case of a cache hit, cache controller 136 provides the requested data from cache 132 to cache 102, which satisfies the memory access request as described above. If a cache miss occurs in cache 132, cache controller 136 provides the memory access request to memory 140. Memory controller 145 fetches the cache line at the memory address of memory 140 targeted by the request, provides the cache line to cache 132, and cache 132 provides the cache line to cache 102, and the memory access request is satisfied. Thus, as described above, the memory access request traverses the memory hierarchy until the requested data is found. The requested data is then transferred to cache 102, and the memory access request is satisfied.

[0027] In some embodiments, each of cache 102 and cache 132 is a set-associative cache, and each cache is divided into a number of sets. Each set includes a number of ways, and each way corresponds to a cache entry that stores a cache line. Each set stores only cache lines associated with a subset of memory addresses, and the subset associated with the set is identified by the corresponding cache controller based on a portion of the memory address called an index. By employing set-associative, caches 102, 132 facilitate a relatively quick identification of cache misses and cache hits.

[0028] In some embodiments, the cache 102 is sized such that it typically cannot store all data requested or potentially requested by the processor 108 at a given time, thereby requiring that the data be transferred across the memory hierarchy as described above. To ensure data coherence and efficient data transfer across the memory hierarchy, the cache controller 104 identifies whether there is an entry in the set available to store an incoming cache line, and if not, implements a replacement policy to select one of the entries in the set to replace it. The availability of a cache entry is indicated by state information associated with the entry, called the entry's valid state. In particular, a cache line with an invalid state (referred to herein as an invalid cache line) is a cache line available to store data, and a cache line with a valid valid state (referred to herein as a valid cache line) is a cache line that is not available to store data unless replaced. To replace a valid cache line in an entry with an incoming cache line, the cache controller for the cache first evicts the valid cache line by transferring the valid cache line to one or more other levels of the memory hierarchy, and then stores the incoming cache line in the entry.

[0029] For example, in response to a reset of the processing system 100, all cache lines in the cache 102 are set to an invalid state by the cache controller 104. When a cache entry is populated in a cache line retrieved from memory 140, the cache controller 104 sets the cache entry to an enabled state. A cache way containing a cache line set to an invalid state may receive an incoming cache line that will displace or overwrite the invalid cache line. When the cache 102 receives a cache line to be stored, it must select a cache way in which the cache line will be stored. If the cache set associated with the incoming cache line has available space (i.e., has one or more cache ways indicated as containing an invalid cache line), the incoming cache line is stored in one of the invalid ways. However, if all cache ways in the set associated with the incoming cache line are indicated as enabled, the cache controller 104 selects a cache line in the set associated with the newly retrieved cache line to create space for the incoming cache line.

[0030] The specific criteria used by the cache controller 104 to select which cache line to replace are called replacement policies. For example, in some embodiments, the cache controller 104 implements a cache replacement policy 112 in the cache 102, which selects for eviction the longest unused cache line in the cache set associated with the incoming cache line (i.e., the cache line that has been the target of memory access operations for the longest time). In other embodiments, the cache controller 104 implements a cache replacement policy 112 in the cache 102, which selects for eviction the least used cache line in the cache set associated with the incoming cache line (i.e., the cache line that has historically been the least used as a target of memory access operations over a particular period). Other replacement policies include rereference interval predictions.

[0031] Access via the non-local memory channel 116 will take longer than access via the local memory channel 106 because it must traverse a longer distance and may encounter delays, for example, on the bus 118. Variable caching latency resulting from the difference in access time for cache misses satisfied via local versus non-local memory channels negatively impacts processing performance. In some embodiments, the cache controller 104 includes a measurement circuit 114. The measurement circuit 114 periodically collects information regarding the ratio of local data to non-local data stored in the cache 102. For example, in some embodiments, the measurement circuit 114 maintains a count of the number of cache lines present in the cache 102 fetched via the local memory channel 106 and a count of the number of cache lines present in the cache 102 fetched via the non-local memory channel 116.

[0032] To facilitate more efficient caching of both local and non-local data in cache 102, the cache controller 104 applies a cache replacement policy 112, at least partially based on the locality of the data stored in cache 102 (i.e., local vs. non-local origin). Depending on factors such as cache access history, requests from applications (not shown) running on processor 108, and the ratio of local data to non-local data stored in cache 102, the cache controller 104 assigns a priority to each cache line based on whether the cache line was accessed via local memory channel 106 or non-local memory channel 116, as will be described in more detail below.

[0033] In some embodiments, the processing system 100 is implemented in a multichiplet system as shown in Figure 2. Figure 2 is a block diagram of the processing system 200, which includes multiple processing units partitioned into subsets of multiple dies, according to some embodiments. The processing system 100 implements a multidie processing unit 201 according to some embodiments. In various embodiments, the multidie processing unit 201 is a parallel processor that includes any cooperating collection of hardware and / or software that performs functions and calculations associated with accelerating graphics processing tasks, data parallel tasks, nested data parallel tasks, etc., in an accelerated manner in relation to resources such as conventional CPUs, conventional graphics processing units (GPUs), and combinations thereof.

[0034] The processing system 200 includes one or more central processing units (CPUs) 250. Although one CPU 250 is shown in Figure 1, some embodiments of the processing system 100 include more CPUs. The bus 245 supports data flow between endpoints within the processing system 200, including the multi-die processing unit 201 and the CPUs 250, as well as between other endpoints.

[0035] In various embodiments, the CPU 250 is connected via bus 245 to memories 270, 272, and 274, which in some embodiments are implemented as dynamic random access memory (DRAM). In various embodiments, memories 270, 272, and 274 may also be implemented using other types of memory, including static random access memory (SRAM), non-volatile RAM, etc. In the illustrated embodiment, the CPU 250 also communicates with memories 270, 272, and 274 and the multi-die processing unit 201 via bus 245. However, some embodiments of the processing system 200 include a multi-die processing unit 201 that communicates with the CPU 250 either by direct connection or via a dedicated bus, bridge, switch, router, etc.

[0036] As shown, the CPU 250 includes several processes, such as running one or more applications 260 for generating graphics commands. In various embodiments, one or more applications 260 include applications that utilize the functionality of the multi-die processing unit 201, such as applications that generate work in the processing system 200 or operating system (OS). In some embodiments, the application 260 includes one or more graphics instructions that instruct the multi-die processing unit 201 to render a graphical user interface (GUI) and / or a graphics scene. For example, in some embodiments, the graphics instructions include instructions that define one or more sets of graphics primitives to be rendered by the multi-die processing unit 201.

[0037] In some embodiments, application 260 utilizes a graphics application programming interface (API) 265 to invoke a user-mode driver (not shown) (or a similar GPU driver). The user-mode driver issues one or more commands to the multi-die processing unit 201 to render one or more graphics primitives into a displayable graphics image. Based on the graphics instructions issued by application 260 to the user-mode driver, the user-mode driver generates one or more graphics commands that specify one or more actions of the multi-die processing unit 201 to perform graphics rendering. In some embodiments, the user-mode driver is part of application 260 running on CPU 250. For example, in some embodiments, the user-mode driver is part of a game application running on CPU 250. Similarly, in some embodiments, a kernel-mode driver (not shown), either alone or in combination with the user-mode driver, generates one or more graphics commands as part of an operating system running on CPU 250.

[0038] The multi-die processing unit 201 shown in Figure 2 includes three chiplet sets 202, 203, and 204. Each chiplet set 202, 203, and 204 includes a set of shader engine (SE) dies 205 used to receive and execute commands simultaneously or in parallel. In some embodiments, each SE die 205 includes a configurable number of shader engines, each shader engine includes a configurable number of workgroup processors, and each workgroup processor includes a configurable number of compute units. Some embodiments of the SE die 205 are configured to shade the vertices of primitives representing a model of the scene using information in draw calls received from the CPU 250. While three SE dies 205 are shown for each GPU chiplet set 202, 203, and 204, as shown in Figure 2, some embodiments of the multi-die processing unit 201 include more or fewer shader engine dies 205. Furthermore, although three L2 cache dies 210, 220, and 230 are shown in Figure 2, the number of L2 cache dies and corresponding shader engine dies within the multi-die processing unit 201 is a matter of design choice and will vary in other embodiments.

[0039] Each SE die 205 includes an L1 cache 208 for storing data directly accessed by the corresponding SE die 205. In addition, each set of SE dies 205 in chiplet sets 202, 203, and 204 is connected to an L2 cache die (e.g., L2 cache die 210, L2 cache die 220, and L2 cache die 230) which includes corresponding L2 caches 212, 222, and 232 for storing frequently used data and instructions, including both local and non-local data for access by the shader engine in each SE die 205, and corresponding L2 cache controllers 214, 224, and 234. In some embodiments, each L2 cache 212, 222, and 232 is connected to one or more L1 caches 208 implemented on the SE die 205 and one or more L3 caches (or other final-level caches) implemented on the processing system 200. Collectively, the caches form a cache hierarchy. Each L2 cache die 210, 220, and 230 is further connected to its respective off-die memory 270, 272, and 274 via its respective local memory channels 275 and 226.

[0040] In the embodiment shown in Figure 2, the coherent data fabric 240 connects the L2 cache dies 210, 220, and 230 in a communicative manner. The coherent data fabric 240 includes a crossbar switch for routing memory access requests and memory responses to any memory access. In some embodiments, the coherent data fabric 240 also includes a system memory map defined by a basic input / output system (BIOS) for determining the destination of memory accesses based on the system configuration, as well as buffers for each virtual connection.

[0041] As a general overview of operation, L2 cache 212 is communicatively coupled to each of the L1 caches 208 on the corresponding SE dies 205 of chiplet set 202 via data bus 206, and L2 cache die 210 is communicatively coupled to memory 270 via local memory channel 275. Similarly, L2 cache 222 is communicatively coupled to each of the L1 caches 208 on the corresponding SE dies 205 of chiplet set 204, L2 cache 232 is communicatively coupled to each of the L1 caches 208 on the corresponding SE dies 205 of chiplet set 203 via their respective data buses, and L2 cache die 220 is communicatively coupled to memory 272 via local memory channel 226. L2 cache 212 is communicatively coupled to each of the L1 caches 208 on the SE dies 205 of chiplet sets 203 and 204 via non-local memory channel 216 which accesses the coherent data fabric 240. Similarly, the L2 caches 222 and 232 are communicably coupled to the L1 cache 208 of the SE die 205 of chiplet set 202 (and to the other of chiplet sets 203 and 204) via their respective non-local memory channels. Thus, any inter-chiplet set memory access is routed via the coherent data fabric 240 as needed to access memory channels on other chiplet sets. In this way, the multi-die processing unit 201 includes chiplet sets that can be addressed as a single monolithic processing unit from the perspective of a software developer (for example, the CPU 250 and any associated applications / drivers are unaware of the chiplet-based architecture), thus avoiding the need for any chiplet-specific considerations on the programmer's or developer's side.

[0042] However, any memory access between chiplet sets incurs a latency penalty because it must traverse non-local memory channels. For example, in response to a cache miss of non-local data in memory 270 connected to the L2 cache die 210 via L2 cache 212 and local memory channel 275, the L2 cache controller 214 provides a memory access request to the L2 cache controller 224 in chiplet set 203 via the coherent data fabric 240. The L2 cache controller 224 receives the memory access request from the L2 cache controller 214. Upon receiving the memory access request, the L2 cache controller 224 identifies whether an entry in the L2 cache 222 stores the data associated with the memory address targeted by the memory access request. In case of a cache hit, the L2 cache controller 224 provides the requested data from the L2 cache 222 to the L2 cache 212 via the coherent data fabric 240, which satisfies the memory access request. If a cache miss occurs in the L2 cache 222, the L2 cache controller 224 provides a memory access request to memory 272 via a memory channel 226 that is not local to the L2 cache die 210 where the cache miss occurred. The memory controller (not shown) retrieves a cache line at the memory address of memory 272 targeted by the request and provides the cache line to the L2 cache 232, which then provides the cache line to the L2 cache 212, thus satisfying the memory access request.

[0043] Figure 3 is a block diagram illustrating the prioritization of local and non-local data based on a cache replacement policy and the ratio of local data to non-local data stored in the cache, according to several embodiments. When a non-random replacement policy is used, the cache effectively maintains an ordered list of eviction candidates, which may be called a reference chain of entries, where accessed entries are promoted to the head of the chain according to the replacement policy, in the case of LRU or MRU type policies. Under an LRU policy, entries are evicted from the tail of the chain, and under an MRU policy, entries are evicted from the head of the chain.

[0044] In the illustrated example, the L2 cache 212 contains a reference chain of cache entries. Under a cache replacement policy that dislikes both local and non-local data, data fetched into the L2 cache 212 is initially placed in the middle of the reference chain. If data is subsequently requested, it moves toward the head 301 of the reference chain. When additional data is fetched into the L2 cache 212, the data initially stored in the middle of the reference chain moves toward the bottom of the reference chain, called the tail 302 of the reference chain, if not requested. When data is evicted from the L2 cache 212, the data is replaced from the tail 302 of the reference chain. Under the cache replacement policy 300, the L2 cache controller 214 assigns all data 322 (including both local and non-local data) to the same priority in the reference chain when it is first fetched into the L2 cache 212.

[0045] Under the cache replacement policy 310, non-local data 324 is preferred to reduce average latency. Based on the cache replacement policy 310, the L2 cache controller 214 assigns local data 326 priority to be placed in the middle of the reference chain if it is fetched first, and non-local data 324 priority to be placed towards the head 301 of the reference chain. Therefore, local data 326 is more likely to reach the tail 302 of the reference chain before non-local data 324, and is therefore more likely to be evicted from the L2 cache 212 than non-local data 324.

[0046] Conversely, under the cache substitution policy 320, local data 322 is preferred. For example, if the miss rates between local data 322 and non-local data 324 are non-uniform, the placement of local data can be preferred. For example, if the re-referencing interval for non-local data 324 is such that it always thrashes the cache, but a shorter re-referencing interval for local data 322 keeps the data in the cache, the cache substitution policy 320 may preferentially place local data 322. Based on the cache substitution policy 320, the L2 cache controller 214 assigns a priority to non-local data 324 to place it in the middle of the reference chain if it is fetched first, and a priority to place local data 326 toward the head 301 of the reference chain. Therefore, non-local data 324 is more likely to reach the tail 302 of the reference chain before local data 326, and is therefore more likely to be evicted from the L2 cache 212 than local data 326. Therefore, the assigned priority of local and non-local data is represented by the initial proximity of the data to the head 301 or tail 302 of the L2 cache 212.

[0047] In some embodiments, the logic in the L2 cache controller 214 applies a cache replacement policy that prioritizes either local or non-local data based on the requests of the application 260 executed in the processing system 200. In some embodiments, the L2 cache controller 214 adjusts the prioritization of local and non-local data based on the relative ratio of local to non-local data stored in the L2 cache 212.

[0048] Figure 4 is a flowchart illustrating a method 400 for adaptively prioritizing local or non-local data based on a cache replacement policy and the ratio of local data to non-local data stored in the cache, according to several embodiments. In some embodiments, the method 400 is implemented in a processing system such as the processing system 100 in Figure 1 or the processing system 200 in Figure 2.

[0049] In block 402, the L2 cache controller 214 applies a cache replacement policy that dislikes both local and non-local data. Therefore, in block 402, the L2 cache controller 214 assigns equal priority to both local data 326 and non-local data 324. In some embodiments, the L2 cache controller 214 therefore places all data 322 in the center of the reference chain of the L2 cache 212.

[0050] In block 404, the measurement circuit 114 of the L2 cache controller 214 periodically collects information about the amount of local data 326 and non-local data 324 stored in the L2 cache 212 (e.g., the number of cache lines). Based on the information collected by the measurement circuit 114, the L2 cache controller 214 determines the relative ratio of local data 326 to non-local data 324 stored in the L2 cache 212.

[0051] In the illustrated example, in block 406, the L2 cache controller 214 determines whether the amount of non-local data 324 stored in the L2 cache 212 exceeds the amount of local data 326 stored in the L2 cache 212 by a first threshold amount. In other embodiments, the L2 cache controller 214 determines whether the ratio of non-local data 324 to local data 326 stored in the L2 cache 212 exceeds a first threshold amount. If the L2 cache controller 214 determines in block 406 that the first threshold has been exceeded, the method flow proceeds to block 410.

[0052] In block 410, the L2 cache controller 214 prioritizes non-local data 324 for substitution. In some embodiments, the L2 cache controller 214 prioritizes non-local data 324 by assigning it a higher priority for substitution. For example, the L2 cache controller 214 assigns local data 326 a priority to place it toward the head 301 of the reference chain of the L2 cache 212, and non-local data 324 a priority to place it toward the middle (or near the tail 302) of the reference chain. The flow of the method then returns to block 404.

[0053] In block 406, if the L2 cache controller 214 determines that the first threshold has not been exceeded, the method flow proceeds to block 408. In block 408, the L2 cache controller 214 determines whether the amount of local data 326 stored in the L2 cache 212 exceeds the amount of non-local data 324 stored in the L2 cache 212 by a second threshold amount. In other embodiments, the L2 cache controller 214 determines whether the ratio of local data 326 to non-local data 324 stored in the L2 cache 212 exceeds the second threshold amount. In some embodiments, the first and second thresholds are the same, and in other embodiments, the first and second thresholds are different from each other. In block 408, if the L2 cache controller 214 determines that the second threshold has not been exceeded, the method flow returns to block 402.

[0054] If the L2 cache controller 214 determines in block 408 that the second threshold has been exceeded, the method flow proceeds to block 412. In block 412, the L2 cache controller 214 prioritizes local data 326 for replacement. In some embodiments, the L2 cache controller 214 prioritizes local data 326 by assigning it a higher priority for replacement. For example, the L2 cache controller 214 assigns non-local data 324 a priority to position local data toward the head 301 of the reference chain of the L2 cache 212, and local data 326 a priority to position local data 326 in the middle (or closer to the tail 302) of the reference chain. The method flow then returns to block 404.

[0055] In some embodiments, the apparatus and techniques described above are implemented in a system including one or more integrated circuit (IC) devices (also referred to as integrated circuit packages or microchips), such as the processing systems described above with reference to Figures 1 to 4. Electronic design automation (EDA) and computer-aided design (CAD) software tools can be used to design and manufacture these IC devices. These design tools are typically represented as one or more software programs. One or more software programs include computer-executable code for operating a computer system to operate with code representing the circuit of one or more IC devices in order to perform at least part of the process of designing or adapting a manufacturing system for manufacturing the circuit. This code may include instructions, data, or combinations of instructions and data. Software instructions representing the design or manufacturing tools are typically stored in a computer-readable storage medium accessible to the computing system. Similarly, code representing one or more stages of designing or manufacturing an IC device is stored in and accessed from the same or different computer-readable storage medium.

[0056] Computer-readable storage media include any non-temporary storage media or combination of non-temporary storage media that are accessible by a computer system during use to provide instructions and / or data to the computer system. Such storage media may include, but are not limited to, optical media (e.g., compact discs (CDs), digital versatile discs (DVDs), Blu-ray® discs), magnetic media (e.g., floppy disks, magnetic tapes, magnetic hard drives), volatile memory (e.g., random access memory (RAM) or cache), non-volatile memory (e.g., read-only memory (ROM) or flash memory), or microelectromechanical system (MEMS) based storage media. Computer-readable storage media (e.g., system RAM or ROM) may be built into the computing system, computer-readable storage media (e.g., magnetic hard drives) may be permanently mounted to the computing system, computer-readable storage media (e.g., optical disks or Universal Serial Bus (USB) based flash memory) may be detachably mounted to the computing system, and computer-readable storage media (e.g., network-accessible storage (NAS)) may be connected to the computer system via a wired or wireless network.

[0057] In some embodiments, certain aspects of the technology described above are implemented by one or more processors of a processing system that executes the software. The software includes one or more sets of executable instructions, which are stored in a non-temporary computer-readable storage medium or otherwise clearly embodied. The software may also include instructions and specific data, which, when executed by one or more processors, operate the one or more processors to execute one or more aspects of the technology described above. Non-temporary computer-readable storage mediums may include, for example, magnetic or optical disk storage devices, solid-state storage devices such as flash memory, caches, random-access memory (RAM), or other non-volatile memory devices (one or more). Executable instructions stored in a non-temporary computer-readable storage medium can be implemented as source code, assembly language code, object code, or other instruction forms that can be interpreted or otherwise executed by one or more processors.

[0058] In addition to the foregoing, it should be noted that not all activities or elements described in the summary are required, and certain activities or parts of devices may not be required, and one or more additional activities may be performed, and one or more additional elements may be included. Furthermore, the order in which the activities are listed does not necessarily indicate the order in which they are performed. Also, the concepts have been described with reference to specific embodiments. However, those skilled in the art will understand that various modifications and variations can be made without departing from the scope of the invention as described in the claims. Therefore, the specification and drawings should be considered illustrative rather than restrictive, and all of these variations are intended to fall within the scope of the invention.

[0059] Benefits, other advantages, and solutions to problems have been described above with respect to specific embodiments. However, benefits, advantages, solutions to problems, and features that may give rise to or manifest any benefits, advantages, or solutions are not to be construed as essential, necessary, or indispensable features to any or all of the claims. Furthermore, the disclosed invention can be modified and implemented in different but similar ways, in ways that are obvious to those skilled in the art who are interested in the teachings of this specification; therefore, the specific embodiments described above are merely illustrative. There are no limitations to the details of the configuration or design shown herein beyond those described in the appended claims. Accordingly, the specific embodiments described above may be modified or altered, and it is clear that all such modifications are within the scope of the disclosed invention. Accordingly, the protection sought herein is described in the appended claims.

Claims

1. It is a method, Based on a cache replacement policy, the system assigns priority to local data stored in the cache of a first subset of chiplets in a processing system comprising a subset of multiple chiplets, and priority to non-local data stored in the cache, wherein the local data includes data accessed from the local memory of the first subset of chiplets via a local memory channel, and the non-local data includes data accessed from the non-local memory of a second subset of chiplets in the processing system via a non-local memory channel. This includes selecting data to be replaced in the cache based on the assigned priority, method.

2. This includes periodically measuring the amount of local data stored in the cache and the amount of non-local data stored in the cache. The method according to claim 1.

3. This includes adjusting the priority of the local data stored in the cache and the priority of the non-local data stored in the cache based on the ratio of local data to non-local data stored in the cache. The method according to claim 1.

4. The priority of the non-local data is increased in accordance with the amount of non-local data stored in the cache, when the amount of local data stored in the cache exceeds a first threshold amount. The method according to claim 1.

5. The process includes increasing the priority of the local data in the event that the amount of local data stored in the cache exceeds the amount of non-local data stored in the cache by a second threshold amount. The method according to claim 1.

6. This includes assigning equal priority to local and non-local data stored in the cache, depending on whether the amount of non-local data stored in the cache does not exceed the amount of local data stored in the cache by a first threshold amount, and whether the amount of local data stored in the cache does not exceed the amount of non-local data stored in the cache by a second threshold amount. The method according to claim 1.

7. The aforementioned assignment is performed based on the application executed in the processing system. The method according to claim 1.

8. A processing system, A cache of the first die of the processing system, configured to store local data accessed from the local memory of a subset of the first die via a local memory channel, and non-local data accessed from the non-local memory of a subset of the second die of the processing system via a non-local memory channel, Equipped with a cache controller, The aforementioned cache controller Based on the cache replacement policy, priority is assigned to the local data and priority is assigned to the non-local data, Based on the assigned priority, select the data to be replaced in the cache, It is configured to do the following: Processing system.

9. The aforementioned cache controller The amount of local data stored in the cache and the amount of non-local data stored in the cache are measured periodically. The processing system of claim 8.

10. The aforementioned cache controller Based on the ratio of local data to non-local data stored in the cache, the priority of the local data stored in the cache and the priority of the non-local data stored in the cache are adjusted. The processing system of claim 8.

11. The aforementioned cache controller The priority of the non-local data is increased when the amount of non-local data stored in the cache exceeds the amount of local data stored in the cache by a first threshold amount. The processing system of claim 8.

12. The aforementioned cache controller The priority of the local data is increased when the amount of local data stored in the cache exceeds the amount of non-local data stored in the cache by a second threshold amount. The processing system of claim 8.

13. The aforementioned cache controller In accordance with the fact that the amount of non-local data stored in the cache does not exceed the amount of local data stored in the cache by a first threshold amount, and the amount of local data stored in the cache does not exceed the amount of non-local data stored in the cache by a second threshold amount, equal priority is assigned to the local data and non-local data stored in the cache. The processing system of claim 8.

14. The aforementioned cache controller Priorities are assigned based on the applications executed in the aforementioned processing system. The processing system of claim 8.

15. A processing system, A cache of a subset of the first die of the processing system, Equipped with a cache controller, The aforementioned cache controller Based on the cache replacement policy and the ratio of local data to non-local data stored in the cache, the system is configured to adaptively prioritize either local or non-local data as the target for replacement in the cache. Local data includes data accessed from the local memory of a subset of the first die via a local memory channel. Non-local data includes data accessed from non-local memory of a subset of the second die of the processing system via a non-local memory channel. Processing system.

16. The aforementioned cache controller The amount of local data stored in the cache and the amount of non-local data stored in the cache are measured periodically. The processing system of claim 15.

17. The aforementioned cache controller The priority of the non-local data is increased when the amount of non-local data stored in the cache exceeds the amount of local data stored in the cache by a first threshold amount. The processing system of claim 15.

18. The aforementioned cache controller The priority of the local data is increased when the amount of local data stored in the cache exceeds the amount of non-local data stored in the cache by a second threshold amount. The processing system of claim 15.

19. The aforementioned cache controller In accordance with the fact that the amount of non-local data stored in the cache does not exceed the amount of local data stored in the cache by a first threshold amount, and the amount of local data stored in the cache does not exceed the amount of non-local data stored in the cache by a second threshold amount, equal priority is assigned to the local data and non-local data stored in the cache. The processing system of claim 15.

20. The aforementioned cache controller Priorities are assigned based on the applications executed in the aforementioned processing system. The processing system of claim 15.

Images